Re: [CentOS] rsync over ssh stalls after completing the job

[Paul Heinlein]

On Wed, 14 Apr 2021, Leon Fauster via CentOS wrote:


On 14.04.21 06:40, Frank Cox wrote:


This doesn't work:
Host *
ForwardX11 yes
host jeff
ForwardX11 no



IMHO - first win. It should be

Host jeff
ForwardX11 no
Host *
ForwardX11 yes


I think that's right. My ssh config has what amounts to four sections:

1. Directives that should not be overridden, ever
2. Host-specific directives
3. Network-specific directives
4. Fall-through defaults

For example:

# = %< =
# don't override
StrictHostKeyChecking ask

# host settings
Host dev.my.net prod.my.net
  ForwardAgent yes
  ForwardX11 yes
  ForwardX11Trusted yes

# network settings
Host *.my.net
  Compression yes
  IdentityFile ~/.ssh/id_ed25519

# defaults
Host *
  Compression no
  ForwardAgent no
  ForwardX11 no
  ForwardX11Trusted no
  Protocol 2
# = %< =

--
Paul Heinlein
heinl...@madboa.com
45.38° N, 122.59° W
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync over ssh stalls after completing the job

[Leon Fauster via CentOS]

On 14.04.21 06:40, Frank Cox wrote:


This doesn't work:
Host *
ForwardX11 yes
host jeff
ForwardX11 no



IMHO - first win. It should be

Host jeff
ForwardX11 no
Host *
ForwardX11 yes

--
Leon
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync over ssh stalls after completing the job

[Frank Cox]

On Tue, 13 Apr 2021 23:36:48 -0400
Chris Schanzle via CentOS wrote:

> I discovered if I set ForwardX11=no (either on ssh command line or in
> ~/.ssh/config) the hang does not happen.  But why does that matter?  No
> updates to openssh.

Far out!  That's the solution!

Well, not really a solution but it's certainly a work-around that makes rsync 
over ssh work without hanging.

I discovered that you have to put the "ForwardX11 no" into the default part of 
.ssh/config.

It doesn't work if you just specify the host involved.

This doesn't work:
Host *
ForwardX11 yes
host jeff
ForwardX11 no

This does work:
Host *
ForwardX11 no

Someone should probably file a bug report about this, but I have no idea what 
package it pertains to.  As you said, openssh wasn't updated at the point that 
this broke.

If it's of any value, here's a list of what was updated last on this computer.  
The ones that look most suspicious to me would be kernel, crypto-policies 
and/or systemd.

Packages Altered:
Install  kernel-4.18.0-240.22.1.el8_3.x86_64@baseos
Install  kernel-core-4.18.0-240.22.1.el8_3.x86_64   @baseos
Install  kernel-debug-devel-4.18.0-240.22.1.el8_3.x86_64@baseos
Install  kernel-devel-4.18.0-240.22.1.el8_3.x86_64  @baseos
Install  kernel-modules-4.18.0-240.22.1.el8_3.x86_64@baseos
Upgrade  dbus-x11-1:1.12.8-12.el8_3.x86_64  
@appstream
Upgraded dbus-x11-1:1.12.8-11.el8.x86_64@@System
Upgrade  flatpak-1.6.2-6.el8_3.x86_64   
@appstream
Upgraded flatpak-1.6.2-5.el8_3.x86_64   @@System
Upgrade  flatpak-libs-1.6.2-6.el8_3.x86_64  
@appstream
Upgraded flatpak-libs-1.6.2-5.el8_3.x86_64  @@System
Upgrade  flatpak-selinux-1.6.2-6.el8_3.noarch   
@appstream
Upgraded flatpak-selinux-1.6.2-5.el8_3.noarch   @@System
Upgrade  flatpak-session-helper-1.6.2-6.el8_3.x86_64
@appstream
Upgraded flatpak-session-helper-1.6.2-5.el8_3.x86_64@@System
Upgrade  java-1.8.0-openjdk-1:1.8.0.282.b08-2.el8_3.x86_64  
@appstream
Upgraded java-1.8.0-openjdk-1:1.8.0.275.b01-1.el8_3.x86_64  @@System
Upgrade  java-1.8.0-openjdk-devel-1:1.8.0.282.b08-2.el8_3.x86_64
@appstream
Upgraded java-1.8.0-openjdk-devel-1:1.8.0.275.b01-1.el8_3.x86_64@@System
Upgrade  java-1.8.0-openjdk-headless-1:1.8.0.282.b08-2.el8_3.x86_64 
@appstream
Upgraded java-1.8.0-openjdk-headless-1:1.8.0.275.b01-1.el8_3.x86_64 @@System
Upgrade  bpftool-4.18.0-240.22.1.el8_3.x86_64   @baseos
Upgraded bpftool-4.18.0-240.15.1.el8_3.x86_64   @@System
Upgrade  crypto-policies-20210209-1.gitbfb6bed.el8_3.noarch @baseos
Upgraded crypto-policies-20200713-1.git51d1222.el8.noarch   @@System
Upgrade  crypto-policies-scripts-20210209-1.gitbfb6bed.el8_3.noarch @baseos
Upgraded crypto-policies-scripts-20200713-1.git51d1222.el8.noarch   @@System
Upgrade  dbus-1:1.12.8-12.el8_3.x86_64  @baseos
Upgraded dbus-1:1.12.8-11.el8.x86_64@@System
Upgrade  dbus-common-1:1.12.8-12.el8_3.noarch   @baseos
Upgraded dbus-common-1:1.12.8-11.el8.noarch @@System
Upgrade  dbus-daemon-1:1.12.8-12.el8_3.x86_64   @baseos
Upgraded dbus-daemon-1:1.12.8-11.el8.x86_64 @@System
Upgrade  dbus-libs-1:1.12.8-12.el8_3.x86_64 @baseos
Upgraded dbus-libs-1:1.12.8-11.el8.x86_64   @@System
Upgrade  dbus-tools-1:1.12.8-12.el8_3.x86_64@baseos
Upgraded dbus-tools-1:1.12.8-11.el8.x86_64  @@System
Upgrade  file-5.33-16.el8_3.1.x86_64@baseos
Upgraded file-5.33-16.el8.x86_64@@System
Upgrade  file-libs-5.33-16.el8_3.1.x86_64   @baseos
Upgraded file-libs-5.33-16.el8.x86_64   @@System
Upgrade  kernel-headers-4.18.0-240.22.1.el8_3.x86_64@baseos
Upgraded kernel-headers-4.18.0-240.15.1.el8_3.x86_64@@System
Upgrade  kernel-tools-4.18.0-240.22.1.el8_3.x86_64  @baseos
Upgraded kernel-tools-4.18.0-240.15.1.el8_3.x86_64  @@System
Upgrade  kernel-tools-libs-4.18.0-240.22.1.el8_3.x86_64 @baseos
Upgraded kernel-tools-libs-4.18.0-240.15.1.el8_3.x86_64 @@System
Upgrade  python3-perf-4.18.0-240.22.1.el8_3.x86_64  @baseos
Upgraded python3-perf-4.18.0-240.15.1.el8_3.x86_64  @@System
Upgrade  

Re: [CentOS] rsync over ssh stalls after completing the job

[Chris Schanzle via CentOS]

On 4/13/21 5:00 PM, Frank Cox wrote:
> On Tue, 13 Apr 2021 22:29:26 +0200
> Simon Matter wrote:
>
>> You could try running strace on the hanging process so see what it's doing.
> [frankcox@mutt temp]$ rsync -avv ../temp/ jeff:temp
> opening connection using: ssh jeff rsync --server -vvlogDtpre.iLsfxC . temp  
> (7 args)
> sending incremental file list
> delta-transmission enabled
> abc is uptodate
> total: matches=0  hash_hits=0  false_alarms=0 data=0
>
> Leaving that sit there apparently doing nothing (but still not giving me my 
> cursor back) I switched to another terminal window and did the following:
>
> [frankcox@mutt ~]$ ps -FA | grep rsync
> frankcox54002435  0 60586  3160   5 14:52 pts/000:00:00 rsync 
> -avv ../temp/ jeff:temp
> frankcox54015400  0 67980  7440   1 14:52 pts/000:00:00 ssh 
jeff rsync --server -vvlogDtpre.iLsfxC . temp
> frankcox55265416  0 55476  1076   3 14:53 pts/100:00:00 grep 
> --color=auto rsync
>
> [frankcox@mutt ~]$ strace -p 5401
> strace: Process 5401 attached
> select(11, [5 9 10], [], NULL, NULL
>
> Then it just sits there with no further action.  I get my cursor back when I 
> hit ctrl-c.
>
> [frankcox@mutt ~]$ strace -p 5400
> strace: Process 5400 attached
> restart_syscall(<... resuming interrupted nanosleep ...>) = 0
> wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0
> nanosleep({tv_sec=0, tv_nsec=2000}, NULL) = 0
> wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0
> nanosleep({tv_sec=0, tv_nsec=2000}, NULL) = 0
> wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0
> nanosleep({tv_sec=0, tv_nsec=2000}, NULL) = 0
> wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0
> nanosleep({tv_sec=0, tv_nsec=2000}, NULL) = 0
> wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0
> nanosleep({tv_sec=0, tv_nsec=2000}, NULL) = 0
> wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0
> nanosleep({tv_sec=0, tv_nsec=2000}, NULL) = 0
> wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0
>
> The wait4-etc line just keeps repeating endlessly until I hit ctrl-c.
>
> Unfortunately, I have no idea what any of the above actually means.  Does it 
> tell us anything interesting?


Yay!  I am glad someone else on the planet is experiencing this.  
I noticed this started happening to me after updating some CentOS Linux 8 
systems today.

I discovered if I set ForwardX11=no (either on ssh command line or in 
~/.ssh/config) the hang does not happen.  But why does that matter?  No updates 
to openssh.

It is not the systemd update doing something silly with session management.  I 
painfully downgraded manually and rebooted to no effect.  
As an aside, why can't we we have nice things in life like 'dnf downgrade 
systemd\*' actually work?  I did the below - might be dumb, but it worked -- 
alternate suggestions to downgrade are appreciated - searching the list and my 
google-fu was off the mark today.

  cd [path-to-repo]/centos/8/BaseOS/x86_64/os/Packages
  dnf downgrade $(rpm -qa systemd\* | grep 239-41.el8_3.2 | sed -e 
's/3\.2/3.1/' -e 's/^/.\//' -e 's/$/.rpm/')

Chris


___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync over ssh stalls after completing the job

[Frank Cox]

On Tue, 13 Apr 2021 22:29:26 +0200
Simon Matter wrote:

> You could try running strace on the hanging process so see what it's doing.

[frankcox@mutt temp]$ rsync -avv ../temp/ jeff:temp
opening connection using: ssh jeff rsync --server -vvlogDtpre.iLsfxC . temp  (7 
args)
sending incremental file list
delta-transmission enabled
abc is uptodate
total: matches=0  hash_hits=0  false_alarms=0 data=0

Leaving that sit there apparently doing nothing (but still not giving me my 
cursor back) I switched to another terminal window and did the following:

[frankcox@mutt ~]$ ps -FA | grep rsync
frankcox54002435  0 60586  3160   5 14:52 pts/000:00:00 rsync -avv 
../temp/ jeff:temp
frankcox54015400  0 67980  7440   1 14:52 pts/000:00:00 ssh jeff 
rsync --server -vvlogDtpre.iLsfxC . temp
frankcox55265416  0 55476  1076   3 14:53 pts/100:00:00 grep 
--color=auto rsync

[frankcox@mutt ~]$ strace -p 5401
strace: Process 5401 attached
select(11, [5 9 10], [], NULL, NULL

Then it just sits there with no further action.  I get my cursor back when I 
hit ctrl-c.

[frankcox@mutt ~]$ strace -p 5400
strace: Process 5400 attached
restart_syscall(<... resuming interrupted nanosleep ...>) = 0
wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0
nanosleep({tv_sec=0, tv_nsec=2000}, NULL) = 0
wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0
nanosleep({tv_sec=0, tv_nsec=2000}, NULL) = 0
wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0
nanosleep({tv_sec=0, tv_nsec=2000}, NULL) = 0
wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0
nanosleep({tv_sec=0, tv_nsec=2000}, NULL) = 0
wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0
nanosleep({tv_sec=0, tv_nsec=2000}, NULL) = 0
wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0
nanosleep({tv_sec=0, tv_nsec=2000}, NULL) = 0
wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0

The wait4-etc line just keeps repeating endlessly until I hit ctrl-c.

Unfortunately, I have no idea what any of the above actually means.  Does it 
tell us anything interesting?


-- 
MELVILLE THEATRE ~ Real D 3D Digital Cinema ~ www.melvilletheatre.com
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync over ssh stalls after completing the job

[Simon Matter]

> On Tue, 13 Apr 2021 12:09:42 -0700 (PDT)
> Paul Heinlein wrote:
>
>> Is there any chance that your shell is configured to emit anything to
>> stderr or stdout when you logout of jeff? It's fairly rare, but I've
>> seen logout messages mess up rsync before.
>
> I don't think so.  The only change from the default .bashrc on both
> machines is the addition of "unset command_not_found_handle", and in
> .bash_profile on mutt I have xmodmap -e "keycode 135 = 0x", which
> hasn't changed since 2016 according to the timestamp on the file.  Other
> than that, the .bash* files are just the defaults.  sshd_config has
> nothing other than the default settings for logout messages, and in any
> event none of these things have changed since last week and it did work
> then.
>
> I wonder if it's something to do with the last Centos 8 update.  There was
> a fair amount of stuff updated including the kernel.  I just rebooted both
> machines with the previous kernel and nothing changed, so that doesn't
> appear to be it either.
>

You could try running strace on the hanging process so see what it's doing.

Regards,
Simon

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync over ssh stalls after completing the job

[Frank Cox]

On Tue, 13 Apr 2021 12:09:42 -0700 (PDT)
Paul Heinlein wrote:

> Is there any chance that your shell is configured to emit anything to 
> stderr or stdout when you logout of jeff? It's fairly rare, but I've 
> seen logout messages mess up rsync before.

I don't think so.  The only change from the default .bashrc on both machines is 
the addition of "unset command_not_found_handle", and in .bash_profile on mutt 
I have xmodmap -e "keycode 135 = 0x", which hasn't changed since 2016 
according to the timestamp on the file.  Other than that, the .bash* files are 
just the defaults.  sshd_config has nothing other than the default settings for 
logout messages, and in any event none of these things have changed since last 
week and it did work then.

I wonder if it's something to do with the last Centos 8 update.  There was a 
fair amount of stuff updated including the kernel.  I just rebooted both 
machines with the previous kernel and nothing changed, so that doesn't appear 
to be it either.


-- 
MELVILLE THEATRE ~ Real D 3D Digital Cinema ~ www.melvilletheatre.com
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync over ssh stalls after completing the job

[Paul Heinlein]

On Tue, 13 Apr 2021, Frank Cox wrote:


Here's a weird one.

I have two Centos 8 machines that use rsync-over-ssh to back up files between 
each other.  (Each machine acts as a backup machine for the other one.)

There's are nightly cronjobs that do the backing up, the commands look like 
this:

rsync -av --delete /home/mydirectory jeff:/home/mydirectorybackup

That command works fine when it's run through the cronjob.

When I try to run a rsync command between mutt and jeff from the 
commandline, that's where the problem starts.  It worked a few days 
ago but now when I log into jeff and do a rsync to or from mutt it 
works fine.  When I log into mutt and do a rsync to or from jeff it 
works and does the job, but then it seems to stall afterward and I 
have to hit ctrl-c to get my cursor back.


Is there any chance that your shell is configured to emit anything to 
stderr or stdout when you logout of jeff? It's fairly rare, but I've 
seen logout messages mess up rsync before.


--
Paul Heinlein
heinl...@madboa.com
45.38° N, 122.59° W
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync over ssh stalls after completing the job

[Frank Cox]

On Tue, 13 Apr 2021 13:07:19 -0500
Christopher Wensink wrote:

> Try using --whole-file / -W

Using that command, it says "delta-transmission disabled for local transfer or 
--whole-file
", but it still stalls at the end so there's no change.

Since it works fine transferring files over nfs, it has to be something about 
the interaction with ssh.  (The fact that it also works fine with ssh as a 
cronjob adds to the weirdness, of course.)

Reading up a bit on what can cause ssh to appear to stall, I just un-commented 
"useDNS no" in sshd_config on both machines and restarted sshd.  I think that's 
the default setting anyway, but I made the change and that didn't do anything 
for me either.

I notice, though, that if I ssh from jeff to mutt and type exit, it logs me out 
as soon as I hit enter.  If I ssh from mutt to jeff and type exit, it logs me 
out after a delay of about one second.  I never really noticed if that delay 
was there before or not.  I wonder if it's somehow related.

-- 
MELVILLE THEATRE ~ Real D 3D Digital Cinema ~ www.melvilletheatre.com
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync over ssh stalls after completing the job

[Christopher Wensink]

Try using --whole-file / -W

On 4/13/2021 12:52 PM, Frank Cox wrote:

On Tue, 13 Apr 2021 12:43:16 -0500
Christopher Wensink wrote:


Does it behave any differently when adding a & at the end of the command
when running it manually, or running in a screen session?

Nope.  I get the same stall both ways.  Running it in a screen session looks 
exactly like what I posted earlier, and the & at the end of the command looks 
like this:

[me@mutt temp]$ rsync -avv ../temp/ jeff:temp &
[1] 15694
opening connection using: ssh jeff rsync --server -vvlogDtpre.iLsfxC . temp  (7 
args)
[me@mutt temp]$ sending incremental file list
delta-transmission enabled
bookmarks.html is uptodate
abc
total: matches=38  hash_hits=38  false_alarms=0 data=0
^C

The difference here is that I don't see the "rsync error: received SIGINT, SIGTERM, or 
SIGHUP (code 20) at rsync.c(644) [sender=3.1.3]" after the ctrl-c when I use the &.  
That line shows up only without the &.





--
Christopher Wensink
IS Administrator
Five Star Plastics, Inc
1339 Continental Drive
Eau Claire, WI 54701
Office:  715-831-1682
Mobile:  715-563-3112
Fax:  715-831-6075
cwens...@five-star-plastics.com
www.five-star-plastics.com

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync over ssh stalls after completing the job

[Frank Cox]

On Tue, 13 Apr 2021 12:43:16 -0500
Christopher Wensink wrote:

> Does it behave any differently when adding a & at the end of the command 
> when running it manually, or running in a screen session?

Nope.  I get the same stall both ways.  Running it in a screen session looks 
exactly like what I posted earlier, and the & at the end of the command looks 
like this:

[me@mutt temp]$ rsync -avv ../temp/ jeff:temp &
[1] 15694
opening connection using: ssh jeff rsync --server -vvlogDtpre.iLsfxC . temp  (7 
args)
[me@mutt temp]$ sending incremental file list
delta-transmission enabled
bookmarks.html is uptodate
abc
total: matches=38  hash_hits=38  false_alarms=0 data=0
^C

The difference here is that I don't see the "rsync error: received SIGINT, 
SIGTERM, or SIGHUP (code 20) at rsync.c(644) [sender=3.1.3]" after the ctrl-c 
when I use the &.  That line shows up only without the &.



-- 
MELVILLE THEATRE ~ Real D 3D Digital Cinema ~ www.melvilletheatre.com
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync over ssh stalls after completing the job

[Christopher Wensink]

Does it behave any differently when adding a & at the end of the command 
when running it manually, or running in a screen session?


Chris

On 4/13/2021 11:45 AM, Frank Cox wrote:

Here's a weird one.

I have two Centos 8 machines that use rsync-over-ssh to back up files between 
each other.  (Each machine acts as a backup machine for the other one.)

There's are nightly cronjobs that do the backing up, the commands look like 
this:

rsync -av --delete /home/mydirectory jeff:/home/mydirectorybackup

That command works fine when it's run through the cronjob.

When I try to run a rsync command between mutt and jeff from the commandline, 
that's where the problem starts.  It worked a few days ago but now when I log 
into jeff and do a rsync to or from mutt it works fine.  When I log into mutt 
and do a rsync to or from jeff it works and does the job, but then it seems to 
stall afterward and I have to hit ctrl-c to get my cursor back.

Here's a test run so you can see what happens.

[me@mutt temp]$ rsync -avv ../temp/ jeff:temp
opening connection using: ssh jeff rsync --server -vvlogDtpre.iLsfxC . temp  (7 
args)
sending incremental file list
delta-transmission enabled
bookmarks.html is uptodate
./
abc
total: matches=0  hash_hits=0  false_alarms=0 data=26321
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(644) 
[sender=3.1.3]

A file named bookmarks.html existed in both directories so it wasn't changed, 
and a new file abc was copied to the backup directory.  Then my ctrl-c stopped 
the job and brought the cursor back after it stalled.

scp works fine to copy files either way when logged into either machine and, 
again, my backup-this-to-that cronjobs that run rsync seem to be working fine 
as well.  I just discovered this last night when I went to rsync some files 
manually between these machines. The last time I did that was at least a few 
days ago and it worked fine then.



--
Christopher Wensink
IS Administrator
Five Star Plastics, Inc
1339 Continental Drive
Eau Claire, WI 54701
Office:  715-831-1682
Mobile:  715-563-3112
Fax:  715-831-6075
cwens...@five-star-plastics.com
www.five-star-plastics.com

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

[CentOS] rsync over ssh stalls after completing the job

[Frank Cox]

Here's a weird one.

I have two Centos 8 machines that use rsync-over-ssh to back up files between 
each other.  (Each machine acts as a backup machine for the other one.)

There's are nightly cronjobs that do the backing up, the commands look like 
this:

rsync -av --delete /home/mydirectory jeff:/home/mydirectorybackup

That command works fine when it's run through the cronjob.

When I try to run a rsync command between mutt and jeff from the commandline, 
that's where the problem starts.  It worked a few days ago but now when I log 
into jeff and do a rsync to or from mutt it works fine.  When I log into mutt 
and do a rsync to or from jeff it works and does the job, but then it seems to 
stall afterward and I have to hit ctrl-c to get my cursor back.

Here's a test run so you can see what happens.  

[me@mutt temp]$ rsync -avv ../temp/ jeff:temp
opening connection using: ssh jeff rsync --server -vvlogDtpre.iLsfxC . temp  (7 
args)
sending incremental file list
delta-transmission enabled
bookmarks.html is uptodate
./
abc
total: matches=0  hash_hits=0  false_alarms=0 data=26321
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(644) 
[sender=3.1.3]

A file named bookmarks.html existed in both directories so it wasn't changed, 
and a new file abc was copied to the backup directory.  Then my ctrl-c stopped 
the job and brought the cursor back after it stalled.

scp works fine to copy files either way when logged into either machine and, 
again, my backup-this-to-that cronjobs that run rsync seem to be working fine 
as well.  I just discovered this last night when I went to rsync some files 
manually between these machines. The last time I did that was at least a few 
days ago and it worked fine then.

-- 
MELVILLE THEATRE ~ Real D 3D Digital Cinema ~ www.melvilletheatre.com
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync upgrade

[Simon Matter via CentOS]

> On 8/6/20 12:30 PM, Jack Bailey via CentOS wrote:
>> On 2020-08-06 08:45, J Martin Rushton via CentOS wrote:
>>> You'll need to upgrade to CentOS8.
>>>
>>> C7 is at rsync 3.1.2-10, and will not go above 3.1.2 ever.
>>>
>>> C8.2 is at 3.1.3-7, C8 will always be on 3.1.3
>>>
>>> Martin
>>
>> Another option is to build rsync from source, which is what I did to try
>> out the zstd compression.
>
>
> Just wanted to share Fedora 32's rsync-3.2.2-1.fc32.src.rpm rebuilds
> cleanly without any necessary tweaks on CentOS 7.  I used mock for a clean
> build environment.
>
> It is very empowering to learn how to build your own packages and not very
> hard to get started.  I encourage you to do the same!
>

If you're using 3.2.2, be sure to add this fix to make rsync behave as
expected:

https://github.com/WayneD/rsync/issues/81

Regards,
Simon

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync upgrade

[Gordon Messmer]

On 8/6/20 8:45 AM, J Martin Rushton via CentOS wrote:
C7 is at rsync 3.1.2-10, and will not go above 3.1.2 ever. 



Is there a reason you think that?  RHEL 7 was originally released with 
3.0.9, so we can demonstrate that Red Hat will update not only revision, 
but minor version increases within an RHEL major release.


RHEL 7's last minor release will be 7.9, later this year.  It would 
surprise me if rsync were updated after that, but not terribly 
surprising if that release included an update.


https://access.redhat.com/support/policy/updates/errata/#Maintenance_Support_2_Phase

http://vault.centos.org/7.0.1406/os/Source/SPackages/


___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync upgrade

[Carl George]

FYI, the IUS rsync31u package is going to be retired (unmaintained) in
about a month.  Upstream confirmed that there will never be another
3.1.x release.

https://github.com/WayneD/rsync/issues/80
https://github.com/iusrepo/announce/issues/23

If there is still value in providing a newer rsync package for EL7 in
IUS, I'd recommend requesting an rsync3 package (now that we know that
upstream doesn't maintain old minor versions) on the wishlist.

https://github.com/iusrepo/wishlist

On Thu, Aug 6, 2020 at 6:11 PM John R. Dennison  wrote:
>
> On Thu, Aug 06, 2020 at 10:40:14AM -0500, Christopher Wensink wrote:
> > Can anyone tell me the repository to use to upgrade to a version of
> > rsync later than 3.1.2?
>
> IUS has 3.1.3 for EL7.  More information available via writeup at
> https://wiki.centos.org/AdditionalResources/Repositories or
> https://ius.io/
>
>
>
>
>
> John
>
> --
>  When a man tells you that he got rich through hard work, ask him: "Whose?"
>
> -- Don Marquis (1878-1937), American humorist, journalist, and author
> ___
> CentOS mailing list
> CentOS@centos.org
> https://lists.centos.org/mailman/listinfo/centos



-- 
Carl George

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync upgrade

[Christopher Wensink]

I have not built my own RPMs before, just built from source .tar.gz
files, do you have a good tutorial? 

Is a package built by you inherently more secure than a package strait
from a mirror?



On 8/6/2020 8:12 PM, Chris Schanzle via CentOS wrote:
> On 8/6/20 12:30 PM, Jack Bailey via CentOS wrote:
>> On 2020-08-06 08:45, J Martin Rushton via CentOS wrote:
>>> You'll need to upgrade to CentOS8.
>>>
>>> C7 is at rsync 3.1.2-10, and will not go above 3.1.2 ever.
>>>
>>> C8.2 is at 3.1.3-7, C8 will always be on 3.1.3
>>>
>>> Martin
>> Another option is to build rsync from source, which is what I did to try out 
>> the zstd compression.
>
> Just wanted to share Fedora 32's rsync-3.2.2-1.fc32.src.rpm rebuilds cleanly 
> without any necessary tweaks on CentOS 7.  I used mock for a clean build 
> environment.
>
> It is very empowering to learn how to build your own packages and not very 
> hard to get started.  I encourage you to do the same!
>
>
>
> ___
> CentOS mailing list
> CentOS@centos.org
> https://lists.centos.org/mailman/listinfo/centos
>

-- 
Christopher Wensink
IS Administrator
Five Star Plastics, Inc
1339 Continental Drive 
Eau Claire, WI 54701
Office:  715-831-1682
Mobile:  715-563-3112
Fax:  715-831-6075
cwens...@five-star-plastics.com
www.five-star-plastics.com

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync upgrade

[Chris Schanzle via CentOS]

On 8/6/20 12:30 PM, Jack Bailey via CentOS wrote:
> On 2020-08-06 08:45, J Martin Rushton via CentOS wrote:
>> You'll need to upgrade to CentOS8.
>>
>> C7 is at rsync 3.1.2-10, and will not go above 3.1.2 ever.
>>
>> C8.2 is at 3.1.3-7, C8 will always be on 3.1.3
>>
>> Martin
>
> Another option is to build rsync from source, which is what I did to try out 
> the zstd compression.


Just wanted to share Fedora 32's rsync-3.2.2-1.fc32.src.rpm rebuilds cleanly 
without any necessary tweaks on CentOS 7.  I used mock for a clean build 
environment.

It is very empowering to learn how to build your own packages and not very hard 
to get started.  I encourage you to do the same!



___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync upgrade

[Christopher Wensink]

I used that one, and it fixed the issue with the synology backup.  4 hrs
12 min into the backup and 1.5 TB backed up. 

Thanks everyone!

On 8/6/2020 6:11 PM, John R. Dennison wrote:
> On Thu, Aug 06, 2020 at 10:40:14AM -0500, Christopher Wensink wrote:
>> Can anyone tell me the repository to use to upgrade to a version of
>> rsync later than 3.1.2?
> IUS has 3.1.3 for EL7.  More information available via writeup at
> https://wiki.centos.org/AdditionalResources/Repositories or 
> https://ius.io/
>
>
>
>
>
>   John
>
>
> ___
> CentOS mailing list
> CentOS@centos.org
> https://lists.centos.org/mailman/listinfo/centos

-- 
Christopher Wensink
IS Administrator
Five Star Plastics, Inc
1339 Continental Drive 
Eau Claire, WI 54701
Office:  715-831-1682
Mobile:  715-563-3112
Fax:  715-831-6075
cwens...@five-star-plastics.com
www.five-star-plastics.com

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync upgrade

[John R. Dennison]

On Thu, Aug 06, 2020 at 10:40:14AM -0500, Christopher Wensink wrote:
> Can anyone tell me the repository to use to upgrade to a version of
> rsync later than 3.1.2?

IUS has 3.1.3 for EL7.  More information available via writeup at
https://wiki.centos.org/AdditionalResources/Repositories or 
https://ius.io/





John

-- 
 When a man tells you that he got rich through hard work, ask him: "Whose?"

-- Don Marquis (1878-1937), American humorist, journalist, and author


pgpRXZcw1fkSz.pgp
Description: PGP signature
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync upgrade

[Leon Fauster via CentOS]

Am 06.08.20 um 21:10 schrieb Phil Perry:

On 06/08/2020 17:30, Jack Bailey via CentOS wrote:

On 2020-08-06 08:45, J Martin Rushton via CentOS wrote:

You'll need to upgrade to CentOS8.

C7 is at rsync 3.1.2-10, and will not go above 3.1.2 ever.

C8.2 is at 3.1.3-7, C8 will always be on 3.1.3

Martin


Another option is to build rsync from source, which is what I did to 
try out the zstd compression.




Or you could try rebuilding the latest fedora SRPM on el7 (or the RHEL8 
SRPM if that is new enough for you). Sometimes it's straight forward, 
other times this approach may fail with newer build requirements than 
those provided by el7.


Worth a try though.



or

https://rsync.samba.org/download.html#Binaries
https://download.samba.org/pub/rsync/binaries/centos-7.8-x86_64/

or

asking upstream to upgrade:

https://bugzilla.redhat.com


or

As they always say to me:

"If this issue is critical or in any way time sensitive, please
raise a ticket through the regular Red Hat support channels to ensure
it receives the proper attention and prioritization to assure a timely
resolution. For information on how to contact the Red Hat production
support team, please visit: https://access.redhat.com/support;



As "Hick's law" say https://en.wikipedia.org/wiki/Hick's_law

--

After the boot hole scream in the funny mode now,

Leon :)






___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync upgrade

[Christopher Wensink]

The problem that I keep running into is, our synology backup device
(RS3618xs) has an option to make a connection to each Linux VM, and our
file server's backup keeps failing every night with the error message: 

"The samba shared folder settings of the source data cannot support
backup.  Please disable "store dos attributes" and "vss objects".

In smb.conf the only reference to store dos attributes is a commented
out line, and nothing references vss objects. 

This error is coming up on a new vm running Centos 7.8.2003 (Core),
Samba 4.10.4, rsync 3.1.2.

The exact same setup on the old vm running Centos6.10 (Final), Samba
3.6.23-53.el6_10, rsync 3.0.6 with the same files which have DOS
attributes on them, and the backup works fine. 

Other VM's running the same command structure using the same version of
Centos 7 work without issues. 

Per Synology support the issue is the files that contain DOS attributes
are breaking the backup.

Is anyone aware of issues with files that contain DOS attributes
breaking rsync backups on Centos7.8.2003 with Samba 4.10.4 and Rsync 3.1.2?

Chris


On 8/6/2020 2:10 PM, Phil Perry wrote:
> On 06/08/2020 17:30, Jack Bailey via CentOS wrote:
>> On 2020-08-06 08:45, J Martin Rushton via CentOS wrote:
>>> You'll need to upgrade to CentOS8.
>>>
>>> C7 is at rsync 3.1.2-10, and will not go above 3.1.2 ever.
>>>
>>> C8.2 is at 3.1.3-7, C8 will always be on 3.1.3
>>>
>>> Martin
>>
>> Another option is to build rsync from source, which is what I did to
>> try out the zstd compression.
>>
>
> Or you could try rebuilding the latest fedora SRPM on el7 (or the
> RHEL8 SRPM if that is new enough for you). Sometimes it's straight
> forward, other times this approach may fail with newer build
> requirements than those provided by el7.
>
> Worth a try though.
>
> ___
> CentOS mailing list
> CentOS@centos.org
> https://lists.centos.org/mailman/listinfo/centos

-- 
Christopher Wensink
IS Administrator
Five Star Plastics, Inc
1339 Continental Drive 
Eau Claire, WI 54701
Office:  715-831-1682
Mobile:  715-563-3112
Fax:  715-831-6075
cwens...@five-star-plastics.com
www.five-star-plastics.com

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync upgrade

[Phil Perry]

On 06/08/2020 17:30, Jack Bailey via CentOS wrote:

On 2020-08-06 08:45, J Martin Rushton via CentOS wrote:

You'll need to upgrade to CentOS8.

C7 is at rsync 3.1.2-10, and will not go above 3.1.2 ever.

C8.2 is at 3.1.3-7, C8 will always be on 3.1.3

Martin


Another option is to build rsync from source, which is what I did to try 
out the zstd compression.




Or you could try rebuilding the latest fedora SRPM on el7 (or the RHEL8 
SRPM if that is new enough for you). Sometimes it's straight forward, 
other times this approach may fail with newer build requirements than 
those provided by el7.


Worth a try though.

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync upgrade

[Jack Bailey via CentOS]

On 2020-08-06 08:45, J Martin Rushton via CentOS wrote:

You'll need to upgrade to CentOS8.

C7 is at rsync 3.1.2-10, and will not go above 3.1.2 ever.

C8.2 is at 3.1.3-7, C8 will always be on 3.1.3

Martin


Another option is to build rsync from source, which is what I did to try 
out the zstd compression.


centos7$ rsync --version
rsync  version 3.2.2  protocol version 31
Copyright (C) 1996-2020 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, atimes, batchfiles, inplace,
    append, no ACLs, xattrs, optional protect-args, iconv, symtimes, 
prealloc

Optimizations:
    no SIMD, asm, openssl-crypto
Checksum list:
    xxh64 (xxhash) md5 md4 none
Compress list:
    zstd lz4 zlibx zlib none



On 06/08/2020 16:40, Christopher Wensink wrote:

Can anyone tell me the repository to use to upgrade to a version of
rsync later than 3.1.2?

Chris





___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync upgrade

[J Martin Rushton via CentOS]

You'll need to upgrade to CentOS8.

C7 is at rsync 3.1.2-10, and will not go above 3.1.2 ever.

C8.2 is at 3.1.3-7, C8 will always be on 3.1.3

Martin

On 06/08/2020 16:40, Christopher Wensink wrote:

Can anyone tell me the repository to use to upgrade to a version of
rsync later than 3.1.2?

Chris



--
J Martin Rushton MBCS
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

[CentOS] rsync upgrade

[Christopher Wensink]

Can anyone tell me the repository to use to upgrade to a version of
rsync later than 3.1.2?

Chris

-- 
Christopher Wensink
IS Administrator
Five Star Plastics, Inc
1339 Continental Drive 
Eau Claire, WI 54701
Office:  715-831-1682
Mobile:  715-563-3112
Fax:  715-831-6075
cwens...@five-star-plastics.com
www.five-star-plastics.com

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync: did not see server greeting.

[Ralf Prengel]

Zitat von Ralf Prengel :


Hallo,
until last week I used a debian system for syncing debian and centos  
mirrors. Everything worked fine for years.
After installing centos 7 rsync doesn t work any longer. The  
configuration of the firewall is the same as before.


The message  I see is
rsync: did not see server greeting.

Has anyone a hint for me?
Are ther any main differences between rsync debian and rsync centos?



Hallo,
found it.
We are using a proxy.sh in /etc/profile.d.
This works fine for all other systems but nit with rsync.
Time to read rsync_manpages.

Ralf

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

[CentOS] rsync: did not see server greeting.

[Ralf Prengel]

Hallo,
until last week I used a debian system for syncing debian and centos  
mirrors. Everything worked fine for years.
After installing centos 7 rsync doesn t work any longer. The  
configuration of the firewall is the same as before.


The message  I see is
rsync: did not see server greeting.

Has anyone a hint for me?
Are ther any main differences between rsync debian and rsync centos?

Thanks
Ralf




___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync versioning problem

[Robert Moskowitz]

Warren,

Just want to say thanks.  I have my automated backups going now. Good 
old cronie.


Since I have a cat6 line to my neighbor (as a favor for their net 
access), I only have to move the backup box to his house and I have 
off-site backups for $150 outlay (arm SOC and 4 TB drive) and <$50/yr 
electric (our rate is ~$0.21/KWH).  You can't buy cloud storage for that 
price.


On 08/03/2018 12:17 PM, Warren Young wrote:

On Aug 3, 2018, at 9:59 AM, Robert Moskowitz  wrote:

You haven’t needed "-e ssh” since rsync 2.6.0, which made it the default.  It 
was released in 2004.

How do I specify -p and -l that I cut out of my example?

Add it to ~/.ssh/config:

 Host nevia.htt-consult.com
 Port 
 User rmoskowitz

You might not think of rsync as paying attention to this file, which is 
correct, because it doesn’t.  But since it is executing ssh under the hood, and 
ssh *does* pay attention to that file, it takes effect.

If you’re creating that file for the first time, be sure to chmod 600 it.  Ssh 
will ignore it if it’s not locked-down.  RTFM for details.


/var/flexshare/shares 
x.htt-consult.com:/media/backup/homebase/var/flexshare/shares/

Rsync won’t create multiple levels of directories on the target.  It will only 
create up to one level of missing directories.  Try this:

 $ ssh x.htt-consult.com 'mkdir -p 
/media/backup/homebase/var/flexshare/shares/'

Oh?  I have been doing this in one shape or form for a long time.

Let me clarify:

If only /media/backup exists on the remote machine, I believe you’ll end up 
with /media/backup/shares with that command.

If at least /media/backup/homebase/var/flexshare exists, rsync will always create 
“shares” under “flexshare" on the remote host.

Anything below that level will also be created, regardless of depth.  I’m 
speaking only of the target path given as the last argument to rsync here, not 
of the source machine’s directories discovered during the sync process.

It’s possible this behavior changed since I last looked at it.  I ran into this 
issue many years ago and now ensure that the target path passed to the rsync 
command exists on the remote host before starting the sync.
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos


___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync versioning problem

[Warren Young]

On Aug 3, 2018, at 10:17 AM, Warren Young  wrote:
> 
> It’s possible this behavior changed since I last looked at it.

It looks like the rsync version shipping with CentOS 7 will actually complain 
if multiple target directory levels are missing.  With only ~/tmp existing on 
the target machine, this command:

$ rsync -a path/to/stuff remote-host:tmp/foo/bar

gives:

rsync: mkdir "/home/USER/tmp/foo/bar" failed: No such file or directory (2)
rsync error: error in file IO (code 11) at main.c(657) [Receiver=3.1.2]

I recall a confusing, softer failure mode, so perhaps they decided this would 
be better handled with a hard failure to avoid some end user confusion.  (And 
probably some support email traffic!)

The trailing slash on the source path affects the remote result once you get 
past this issue, but adding a slash doesn’t prevent the error in this condition.
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync versioning problem

[Robert Moskowitz]

Warren, Thanks!

On 08/03/2018 12:17 PM, Warren Young wrote:

On Aug 3, 2018, at 9:59 AM, Robert Moskowitz  wrote:

You haven’t needed "-e ssh” since rsync 2.6.0, which made it the default.  It 
was released in 2004.

How do I specify -p and -l that I cut out of my example?

Add it to ~/.ssh/config:

 Host nevia.htt-consult.com
 Port 
 User rmoskowitz


I will look into this and rtfm  :)

I have been doing this for so long, I never paid attention to the 
change.  The man pages I originally used said to use -e ssh



You might not think of rsync as paying attention to this file, which is 
correct, because it doesn’t.  But since it is executing ssh under the hood, and 
ssh *does* pay attention to that file, it takes effect.

If you’re creating that file for the first time, be sure to chmod 600 it.  Ssh 
will ignore it if it’s not locked-down.  RTFM for details.


And then still ask Dr. Google.  Page's father WAS one of my profs at MSU.




/var/flexshare/shares 
x.htt-consult.com:/media/backup/homebase/var/flexshare/shares/

Rsync won’t create multiple levels of directories on the target.  It will only 
create up to one level of missing directories.  Try this:

 $ ssh x.htt-consult.com 'mkdir -p 
/media/backup/homebase/var/flexshare/shares/'

Oh?  I have been doing this in one shape or form for a long time.

Let me clarify:

If only /media/backup exists on the remote machine, I believe you’ll end up 
with /media/backup/shares with that command.

If at least /media/backup/homebase/var/flexshare exists, rsync will always create 
“shares” under “flexshare" on the remote host.

Anything below that level will also be created, regardless of depth.  I’m 
speaking only of the target path given as the last argument to rsync here, not 
of the source machine’s directories discovered during the sync process.

It’s possible this behavior changed since I last looked at it.  I ran into this 
issue many years ago and now ensure that the target path passed to the rsync 
command exists on the remote host before starting the sync.


Always a challenge to get this right and get the right behavior.  I use 
rsync for backing up my data from my notebook to a USB drive. Also when 
I was active in IEEE 802, at the meetings I could rsync all the 
documents off the local server.  The public server does not support 
rsync.  Plus I rsync all the IETF RFCs and drafts every night.


So I have a lot of old habits.


___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync versioning problem

[Warren Young]

On Aug 3, 2018, at 9:59 AM, Robert Moskowitz  wrote:
> 
>> You haven’t needed "-e ssh” since rsync 2.6.0, which made it the default.  
>> It was released in 2004.
> 
> How do I specify -p and -l that I cut out of my example?

Add it to ~/.ssh/config:

Host nevia.htt-consult.com
Port 
User rmoskowitz

You might not think of rsync as paying attention to this file, which is 
correct, because it doesn’t.  But since it is executing ssh under the hood, and 
ssh *does* pay attention to that file, it takes effect.

If you’re creating that file for the first time, be sure to chmod 600 it.  Ssh 
will ignore it if it’s not locked-down.  RTFM for details.

>>> /var/flexshare/shares 
>>> x.htt-consult.com:/media/backup/homebase/var/flexshare/shares/
>> Rsync won’t create multiple levels of directories on the target.  It will 
>> only create up to one level of missing directories.  Try this:
>> 
>> $ ssh x.htt-consult.com 'mkdir -p 
>> /media/backup/homebase/var/flexshare/shares/'
> 
> Oh?  I have been doing this in one shape or form for a long time.

Let me clarify: 

If only /media/backup exists on the remote machine, I believe you’ll end up 
with /media/backup/shares with that command.

If at least /media/backup/homebase/var/flexshare exists, rsync will always 
create “shares” under “flexshare" on the remote host.

Anything below that level will also be created, regardless of depth.  I’m 
speaking only of the target path given as the last argument to rsync here, not 
of the source machine’s directories discovered during the sync process.

It’s possible this behavior changed since I last looked at it.  I ran into this 
issue many years ago and now ensure that the target path passed to the rsync 
command exists on the remote host before starting the sync.
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync versioning problem

[Robert Moskowitz]

On 08/03/2018 11:07 AM, Warren Young wrote:

On Aug 3, 2018, at 8:57 AM, Robert Moskowitz  wrote:

I seem to have an rsync versioning problem.

Have you ruled out the other causes of that error?  For instance:

 https://askubuntu.com/a/716911


yeah.  It is backups, not backup.  Oops.




And researching this it comes down to a versioning issue.

That seems rather unlikely for such an old and stable tool as rsync, and 
especially for two versions with the same major version number.  If you’d said 
rsync 2 and 3 or we were talking about a tool that still hadn’t hit 1.0 yet, 
I’d believe the protocol was still in flux, but rsync is 22 years old now.

On the other hand, 3.0.6 is nine years old now, so maybe.


rsync -ah --stats --delete -e “ssh"

You haven’t needed "-e ssh” since rsync 2.6.0, which made it the default.  It 
was released in 2004.


How do I specify -p and -l that I cut out of my example?

"ssh -pnnn -luserid"




/var/flexshare/shares 
x.htt-consult.com:/media/backup/homebase/var/flexshare/shares/

Rsync won’t create multiple levels of directories on the target.  It will only 
create up to one level of missing directories.  Try this:

 $ ssh x.htt-consult.com 'mkdir -p 
/media/backup/homebase/var/flexshare/shares/'


Oh?  I have been doing this in one shape or form for a long time. It is 
running right now, copying all the sub dirs and creating any new needed 
with:


rsync -ah --stats --delete -e "ssh -p -l user" 
/var/flexshare/shares/ 
nevia.htt-consult.com:/media/WD3TB01/backups/homebase/var/flexshare/shares


Note the source ending with /

That is needed.  Without it, it creates a shares directory under the 
target directory.

Then retry the rsync.
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos


___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync versioning problem

[Warren Young]

On Aug 3, 2018, at 8:57 AM, Robert Moskowitz  wrote:
> 
> I seem to have an rsync versioning problem.

Have you ruled out the other causes of that error?  For instance:

https://askubuntu.com/a/716911

> And researching this it comes down to a versioning issue.

That seems rather unlikely for such an old and stable tool as rsync, and 
especially for two versions with the same major version number.  If you’d said 
rsync 2 and 3 or we were talking about a tool that still hadn’t hit 1.0 yet, 
I’d believe the protocol was still in flux, but rsync is 22 years old now.

On the other hand, 3.0.6 is nine years old now, so maybe.

> rsync -ah --stats --delete -e “ssh"

You haven’t needed "-e ssh” since rsync 2.6.0, which made it the default.  It 
was released in 2004.

> /var/flexshare/shares 
> x.htt-consult.com:/media/backup/homebase/var/flexshare/shares/

Rsync won’t create multiple levels of directories on the target.  It will only 
create up to one level of missing directories.  Try this:

$ ssh x.htt-consult.com 'mkdir -p 
/media/backup/homebase/var/flexshare/shares/'

Then retry the rsync.
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

[CentOS] rsync versioning problem

[Robert Moskowitz]

I seem to have an rsync versioning problem.

The sender is an old ClearOS6 server with rsynv 3.0.6
The receiver is a brand new Centos7-armv7 server with rsync 3.1.2

I am running rsync over ssh

Got the error:

rsync error: error in rsync protocol data stream (code 12) at io.c(600) 
[sender=3.0.6]


And researching this it comes down to a versioning issue.  But all I 
have found was to upgrade the 3.0.6 system!  That will happen when I 
migrate to ClearOS7!


Is there some option to specify to get this to work?

rsync -ah --stats --delete -e "ssh" /var/flexshare/shares 
x.htt-consult.com:/media/backup/homebase/var/flexshare/shares/



___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync Error: Centos Mirror

[Fabian Arrotin]

On 12/07/17 08:44, Unsub Shafiq wrote:
> Hi,
> 
> I have recently been receiving the following errors when using rsync to
> update my centos local repositories;
> 
> rsync: send_files failed to open "/7/atomic/x86_64/repo/tmp/tmp.HBjcIF"
> (in centos): Permission denied (13)
> rsync: send_files failed to open "/7/atomic/x86_64/repo/tmp/tmp.SWIH0I"
> (in centos): Permission denied (13)
> 
> I have checked and the user that runs the rsync script has all
> permissions to read and write into the destination folder (centos). What
> may be the problem?
> 
> Any help is appreciated.
> 

It was an issue how the tree was generated so it's now fixed at the
build side : https://github.com/CentOS/sig-atomic-buildscripts/pull/284

-- 
Fabian Arrotin
The CentOS Project | http://www.centos.org
gpg key: 56BEC54E | twitter: @arrfab



signature.asc
Description: OpenPGP digital signature
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

[CentOS] Rsync Error: Centos Mirror

[Unsub Shafiq]

Hi,

I have recently been receiving the following errors when using rsync to 
update my centos local repositories;


rsync: send_files failed to open "/7/atomic/x86_64/repo/tmp/tmp.HBjcIF" 
(in centos): Permission denied (13)
rsync: send_files failed to open "/7/atomic/x86_64/repo/tmp/tmp.SWIH0I" 
(in centos): Permission denied (13)


I have checked and the user that runs the rsync script has all 
permissions to read and write into the destination folder (centos). What 
may be the problem?


Any help is appreciated.

--
*Unsub Shafiq*

Inara Technologies (Pvt) Ltd.
Skype: ums.shafiq

Web: www.inaratech.com

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync and cause/source of an empty file

[Gordon Messmer]

On 06/30/2017 01:42 PM, James B. Byrne wrote:

The setup is somewhat complicated in that the VAN
will not permit direct rsync access and so we establish the link via
sshfs and then mount remote location as local.



sshfs doesn't seem to preserve/transmit ctime, so rsync won't be able to 
reliably determine whether a file has changed based on times.  There's a 
small chance that you'll miss updates in this configuration.  That's 
probably not related to your problem, just a note on that setup.


Regarding the empty file, perhaps the source had the file locked for 
reading.  Do you know what type of system is providing the data?


___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync and cause/source of an empty file

[Joseph L. Casale]

-Original Message-
From: CentOS [mailto:centos-boun...@centos.org] On Behalf Of James B.
Byrne
Sent: Friday, June 30, 2017 2:43 PM
To: centos@centos.org
Subject: [CentOS] rsync and cause/source of an empty file
 
> We transfer files from a VAN provider at 15 minute intervals using
> rsync over ssh.  The setup is somewhat complicated in that the VAN
> will not permit direct rsync access and so we establish the link via
> sshfs and then mount remote location as local.
> 
> My question is, given the above conditions and the following rsync
> command:
> 
> /usr/bin/rsync --chmod=o+r --chmod=g+w --itemize-changes
> --remove-sent-files --times /var/spool/imanet/pick_up/*
> /var/spool/imanet/drop_off
> 
> Under what circumstances would a file containing data at the remote
> end (/var/spool/imanet/pick_up/) arrive at our end
> (/var/spool/imanet/drop_off) as an empty file?  No transmission errors
> were logged and multiple files were transferred during the same
> session. All but one arrived with their contents intact.

First guess is you had the misfortune of syncing a file that was
being written to at the same time on the source end.

Not that it helps, but FFS, an EDI provider exposing rsync? What kind
of SP does that? This problem has been so concretely solved ages ago?
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

[CentOS] rsync and cause/source of an empty file

[James B. Byrne]

We transfer files from a VAN provider at 15 minute intervals using
rsync over ssh.  The setup is somewhat complicated in that the VAN
will not permit direct rsync access and so we establish the link via
sshfs and then mount remote location as local.

My question is, given the above conditions and the following rsync
command:

/usr/bin/rsync --chmod=o+r --chmod=g+w --itemize-changes
--remove-sent-files --times /var/spool/imanet/pick_up/* 
/var/spool/imanet/drop_off

Under what circumstances would a file containing data at the remote
end (/var/spool/imanet/pick_up/) arrive at our end
(/var/spool/imanet/drop_off) as an empty file?  No transmission errors
were logged and multiple files were transferred during the same
session. All but one arrived with their contents intact.


-- 
***  e-Mail is NOT a SECURE channel  ***
Do NOT transmit sensitive data via e-Mail
 Do NOT open attachments nor follow links sent by e-Mail

James B. Byrnemailto:byrn...@harte-lyne.ca
Harte & Lyne Limited  http://www.harte-lyne.ca
9 Brockley Drive  vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada  L8E 3C3

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[J Martin Rushton]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Have a coffee or a beer, breathe deeply, then:

On 14/11/15 00:42, Gordon Messmer wrote:
> On 11/13/2015 12:59 PM, J Martin Rushton wrote:
>> Maybe I should have been clearer: use (LVM) OR (RAID1 and
>> break).
> 
> I took your meaning.  I'm saying that's a terrible backup strategy,
> for a list of reasons.
> 
> For instance, it only works if you mirror a single disk.  It
> doesn't work if you use RAID10 or RAID5, or RAID6, or RAIDZ, etc.

That of course is exactly why I said RAID1.

  Breaking RAID
> doesn't make the data consistent, so you might have corrupt files 
> (especially if the system runs any kind of database.  SQL, LDAP,
> etc). It doesn't make the filesystem consistent, so you might have
> a corrupt filesystem.

Possibly, but that is another problem altogether.  Any low level
backup will do the same.  You need to have an understanding of the
filesystem to handle filesystem problems.  Even if the utility
understands the filesystem you have problems with open files such as
databases.

More generally, for anything except a trivial database you should use
the database to dump itself; for instance using mysqldump.  Have a
look at the page
https://mariadb.com/kb/en/mariadb/backup-and-restore-overview/ for (as
it says) an overview.  Try running a database backup timed to complete
before your normal filesystem backups run, whatever method you use.

> 
> Even if you ignore the potential for corruption, you have a backup 
> process that only works on some specific hardware configurations. 
> Everything else has to have a different backup solution.  That's 
> insane.  Use one backup process that works for everything.  You're
> much more likely to consistently back up your data that way.

Remember that this is a last resort if (1) the user can't accept more
sensible backups and handle (or let the backup handle) the dates
safely; (2) the user insists on a snapshot; (3) the user can't use a
filesytem snapshot (ZFS, GPFS etc) and (4) the user can't/won't use
LVM.  You can't refuse to use better solutions" and then complain that
last resort is not as good as the better solutions"!

>> I hope I'm wrong, but you wouldn't be thinking of mounting the
>> broken out copy on a the same system would you?  You must never
>> do that, not even during disaster recovery.  Use dd or similar on
>> the disk, not the mounted partitions - isn't that obvious?  I
>> wasn't trying to give step by step instructions.
> 
> Well, that's *one* of the problems with your advice.  Even if we
> ignore the fact that it doesn't work reliably (and IMO, it
> therefore doesn't work), it's far more complicated than you pretend
> it is.
> 
> Because now you're talking about quiescing your services, breaking
> your RAID, physically removing the drive, connecting it to another
> system, fsck the filesystems, mount them, and backing up the data.
> For each backup.  Every day.

No need to remove if you handle whole disk.  When we used this
technique we only did it monthly - it would be pretty crazy to do
level 0 backups daily.

> Or using 'dd' and... backing up the whole image?  No incremental
> or differentials?

See the previous.

> Your process involves a human being doing physical tasks as part of
> the backup.  Maybe I'm the only one, but I want my backups fully
> automated. People make mistakes.  I don't want them involved in
> regular processes. In fact, the entire point of computing is that
> the computer should do the work so that I don't have to.

See the comments about using better solutions.  I'd be worried though
if you use a solution that doesn't remove the backup media from the
vicinity of the machine.  Fine if you have a remote site, but
otherwise you still need a person to physically take the tapes (or
whatever) out of the machine room to fireproof storage.  That's pretty
manual.

>> Way before LVM existed we used this technique to back up VAXes
>> (and later Alphas) under VMS using "volume shadowing" (ie RAID1).
>> It worked quite happily for several years with disks shared
>> across the cluster. IIRC it was actually recommended by DEC,
>> indeed a selling point, but I don't have any manuals to hand to
>> confirm that nowadays! One thing I did omit was you MUST sync
>> first
> 
> sync flushes the OS data buffers to disk, but it does not sync 
> application data buffers, it does not flush the journal, it doesn't
> make filesystems "clean", and even if you break the RAID volume
> immediately after "sync" there's no guarantee that there weren't
> cached writes from other processes in between those two steps.

The journal is a fair point if it is stored on an separate spindle, as
for instance is possible under XFS.

> There is absolutely no way to make this a reliable process without
> a full shutdown.

Not IME.  At that date the preferred method for monthly backups was a
shutdown and standalone utility for disk-disk copies, but that was not
always possible.  The technique worked.

> 
> 

Re: [CentOS] Rsync and differential Backups

[Mark Milhollan]

On Fri, 13 Nov 2015, Gordon Messmer wrote:

>Breaking a RAID volume doesn't make filesystems consistent, 

While using LVM arranges for some filesystems to be consistent (it is 
not always possible), it does nothing to ensure application consistency 
which can be just as important.  Linux doesn't have a widely deployed 
analog to Windows' VSS, which provides both though only for those that 
cooperate.  On Linux you must arrange to quiesce applications yourself, 
which is seldom possible.

>Breaking the
>RAID will duplicate UUIDs of filesystems and the name of volume groups.  

Making an LVM snapshot duplicates UUIDs (and LABELs) too, the whole LV 
is the same in the snapshot as it was in the source.  There are ways to 
cope with that for XFS (I usually use mount -ro nouuid) -- ext2/3/4 
doesn't care (so just mount -r for them).  If the original filesystem 
isn't yet mounted then a mount by uuid (or label) would not be pretty 
for either.  And that's just two filesystems, others are supported and 
they too will potentially have issues.


/mark
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Gordon Messmer]

On 11/14/2015 03:04 AM, J Martin Rushton wrote:

On 14/11/15 00:42, Gordon Messmer wrote:

For instance, it only works if you mirror a single disk.  It
doesn't work if you use RAID10 or RAID5, or RAID6, or RAIDZ, etc.

That of course is exactly why I said RAID1.


I know.  And I was trying to make the point that the process of breaking 
RAID1 for backup purposes is inflexible in addition to being 
unreliable.  Users should not have to re-engineer their backup system 
for every hardware configuration.




   Breaking RAID

doesn't make the data consistent, so you might have corrupt files
(especially if the system runs any kind of database.  SQL, LDAP,
etc). It doesn't make the filesystem consistent, so you might have
a corrupt filesystem.

Possibly, but that is another problem altogether.  Any low level
backup will do the same.


If you were to attempt a block-level backup of the raw device, then yes, 
you would have similar problems.  But since that is insane, and no one 
is suggesting that process, I didn't feel the need to address it.



   You need to have an understanding of the
filesystem to handle filesystem problems.  Even if the utility
understands the filesystem you have problems with open files such as
databases.


There *are* tools that exist to dump filesystems, but they're not 
intended to be used for backup, and they won't operate on mounted 
filesystems.  For instance, clonezilla includes tools to dump ext4 and 
ntfs filesystems for the purpose of cloning a system.  You could treat 
that as a backup, but you have to shut down the host OS to boot clonezilla.



More generally, for anything except a trivial database you should use
the database to dump itself; for instance using mysqldump.


Uhh no.  I'd argue the opposite.  You should only use a DB dump 
tools for trivial databases (or in some cases, such as PostgreSQL, 
upgrades).  Dumping a database is *slow*.  The only thing slower than 
dumping a database is restoring a database dump.  If you have a 
non-trivial database, you definitely want to quiesce, snapshot, resume, 
and back up the snapshot.



Have a
look at the page
https://mariadb.com/kb/en/mariadb/backup-and-restore-overview/ for (as
it says) an overview.  Try running a database backup timed to complete
before your normal filesystem backups run, whatever method you use.


Again, you seem entirely too willing to accept unreliable processes.  
Timing?  You should absolutely, under no circumstances, trust the timing 
of two processes to not overlap.  If you're dumping data, you should 
either trigger the backup from the dump job, after it completes, or you 
should employ a locking system so that only one of the two processes can 
operate simultaneously.



Remember that this is a last resort if (1) the user can't accept more
sensible backups and handle (or let the backup handle) the dates
safely; (2) the user insists on a snapshot; (3) the user can't use a
filesytem snapshot (ZFS, GPFS etc) and (4) the user can't/won't use
LVM.  You can't refuse to use better solutions" and then complain that
last resort is not as good as the better solutions"!


No one is refusing better solutions.  You are tilting at windmills.


See the comments about using better solutions.  I'd be worried though
if you use a solution that doesn't remove the backup media from the
vicinity of the machine.  Fine if you have a remote site


We agree, there.  You should have backups in a physically separate location.

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Gordon Messmer]

On 11/14/2015 09:01 AM, Mark Milhollan wrote:

On Fri, 13 Nov 2015, Gordon Messmer wrote:

Breaking a RAID volume doesn't make filesystems consistent,

While using LVM arranges for some filesystems to be consistent (it is
not always possible)


Can you explain what you mean?  The standard filesytems, ext4 and XFS, 
both will be made consistent when making an LVM snapshot.



, it does nothing to ensure application consistency
which can be just as important.  Linux doesn't have a widely deployed
analog to Windows' VSS, which provides both though only for those that
cooperate.


I know.  That's why I wrote snapshot:
https://bitbucket.org/gordonmessmer/dragonsdawn-snapshot


On Linux you must arrange to quiesce applications yourself,
which is seldom possible.


I have not found that to be true.  Examples?


Breaking the
RAID will duplicate UUIDs of filesystems and the name of volume groups.

Making an LVM snapshot duplicates UUIDs (and LABELs) too, the whole LV
is the same in the snapshot as it was in the source.


The VG name is the bigger problem.  If you tried to activate the VG in 
the broken RAID1 component, Very Bad Things(TM) would happen.


___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[J Martin Rushton]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 13/11/15 01:52, Benjamin Smith wrote:
> I did exactly this with ZFS on Linux and cut over 24 hours of
> backup lag to just minutes.
> 
> If you're managing data at scale, ZFS just rocks...
> 
> 
> On Tuesday, November 10, 2015 01:16:28 PM Warren Young wrote:
>> On Nov 10, 2015, at 8:46 AM, Gordon Messmer
>> 
> wrote:
>>> On 11/09/2015 09:22 PM, Arun Khan wrote:
 You can use "newer" options of the find command and pass the
 file list
>>> 
>>> the process you described is likely to miss files that are
>>> modified while "find" runs.
>> Well, be fair, rsync can also miss files if files are changing
>> while the backup occurs.  Once rsync has passed through a given
>> section of the tree, it will not see any subsequent changes.
>> 
>> If you need guaranteed-complete filesystem-level snapshots, you
>> need to be using something at the kernel level that can
>> atomically collect the set of modified blocks/files, rather than
>> something that crawls the tree in user space.
>> 
>> On the BSD Now podcast, they recently told a war story about
>> moving one of the main FreeBSD servers to a new data center.
>> rsync was taking 21 hours in back-to-back runs purely due to the
>> amount of files on that server, which gave plenty of time for
>> files to change since the last run.
>> 
>> Solution?  ZFS send:
>> 
>> http://128bitstudios.com/2010/07/23/fun-with-zfs-send-and-receive/
>>
>> 
___
>> CentOS mailing list CentOS@centos.org 
>> https://lists.centos.org/mailman/listinfo/centos
> 
> ___ CentOS mailing
> list CentOS@centos.org 
> https://lists.centos.org/mailman/listinfo/centos
> 
If you really _need_ the guarantee of a snapshot, consider either LVM
or RAID1. Break out a volume from the RAID set, back it up, then
rebuild. If you are paranoid you might want to consider a 3-way RAID1
to ensure you have full shadowing during the backup.  Some commercial
filesystems (such as IBM's GPFS) also include a snapshot command, but
you may need deep pockets.

Other than that, accept as harmless the fact that your backup takes a
finite time.  Provided that you record the time before starting the
sweep, and do the next incremental from that time, then you will catch
all files eventually.  The time lag shouldn't be much though, decent
backup systems scan the sources and generate a work list before
starting to move data.

OT - is ZFS part of the CentOS distro?  I did a quick yum list | grep
- -i zfs and got nothing on a 7.1.1503.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iQIcBAEBAgAGBQJWRbGGAAoJEAF3yXsqtyBlxfoP/2Z7GrmFXz/me/w6Ps9gvdWh
d6wWzwwBUooMjOiUFXspzkXpAIm12cB+M2il4KOutknWDbL5/8iMGgUykoj9Xh7J
lqoG76RJfttdyBomivOqix4Ylx8RMA/SXQlFzJHp7oPowSh8MJpfRXIsWQZCMxWH
8v/gk5ZyGN5Lax9yM6MAyT7YVxh39mYh9+wsP8i3No3UjCLoGBGMEMG75doyl1Tp
zo91dvcYjLiO71zhjg2wG3YJYsxYleJoCHEo/L+2/OgbUi+Pm8JnG0FpinyTbsMm
XkIGcjC+EMS2DQ4rerE+sHnr10N+Z+KJYyk5YX7TEXyID0Vmglfl8ApBMlL44MtK
RhKieM+3KTqmHAwjoQh37RNH9Sfadq130GgPaeHxEWXGQwkaivQaLcge72Cj5r6w
ZCpivejBHpHqyyKnlQcwFwIsoGoWftVJrKZ27tolxIG3A3KiJTGo/uCP5hW/fVjV
giNyRAjfTE8cT5DxOcssIAfjtFwpfx2XxsI4T0p1Hof7S0jYLmCZeQzNGSfMwtvX
oHnXcYg7cer2D4Xfwy9dHGkmqAjMVFUvFMqt6X1EfKfZVtSY3WUAqeouSP57bajZ
KaW/WXJjgp9kF5mKCuS5UsmjnvXJfvmBAcDhzPl3Ut+9Y0oD0/qtfm4Mk9Oyexpn
knL5rfIPygISKma6OTx9
=FM4A
-END PGP SIGNATURE-
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Gordon Messmer]

On 11/13/2015 01:46 AM, J Martin Rushton wrote:

If you really_need_  the guarantee of a snapshot, consider either LVM
or RAID1. Break out a volume from the RAID set, back it up, then
rebuild.


FFS, don't do the latter.  LVM is the standard filesystem backing for 
Red Hat and CentOS systems, and fully supports consistent snapshots 
without doing half-ass shit like breaking a RAID volume.


Breaking a RAID volume doesn't make filesystems consistent, so when you 
try to mount it, you might have a corrupt filesystem, or corrupt data.  
Breaking the RAID will duplicate UUIDs of filesystems and the name of 
volume groups.  There are a whole bunch of configurations where it just 
won't work.  At best, it's unreliable.  Never do this.  Don't advise 
other people to do it.  Use LVM snapshots (or ZFS if that's an option 
for you).

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[J Martin Rushton]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 13/11/15 17:55, Gordon Messmer wrote:
> On 11/13/2015 01:46 AM, J Martin Rushton wrote:
>> If you really_need_  the guarantee of a snapshot, consider either
>> LVM or RAID1. Break out a volume from the RAID set, back it up,
>> then rebuild.
> 
> FFS, don't do the latter.  LVM is the standard filesystem backing
> for Red Hat and CentOS systems, and fully supports consistent
> snapshots without doing half-ass shit like breaking a RAID volume.
> 
> Breaking a RAID volume doesn't make filesystems consistent, so when
> you try to mount it, you might have a corrupt filesystem, or
> corrupt data. Breaking the RAID will duplicate UUIDs of filesystems
> and the name of volume groups.  There are a whole bunch of
> configurations where it just won't work.  At best, it's unreliable.
> Never do this.  Don't advise other people to do it.  Use LVM
> snapshots (or ZFS if that's an option for you). 
> ___ CentOS mailing
> list CentOS@centos.org 
> https://lists.centos.org/mailman/listinfo/centos

Maybe I should have been clearer: use (LVM) OR (RAID1 and break).
Don't use LVM and break, that would be silly.

I hope I'm wrong, but you wouldn't be thinking of mounting the broken
out copy on a the same system would you?  You must never do that, not
even during disaster recovery.  Use dd or similar on the disk, not the
mounted partitions - isn't that obvious?  I wasn't trying to give step
by step instructions.

Way before LVM existed we used this technique to back up VAXes (and
later Alphas) under VMS using "volume shadowing" (ie RAID1). It worked
quite happily for several years with disks shared across the cluster.
IIRC it was actually recommended by DEC, indeed a selling point, but I
don't have any manuals to hand to confirm that nowadays! One thing I
did omit was you MUST sync first (there was an equivalent VMS command,
don't ask me now), and also ensure that as the disks are added back a
full catchup copy occurs.  You may consider it half a mule's
droppings, but it is, after all, what happens if you loose a spindle
and hot replace.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iQIcBAEBAgAGBQJWRk8fAAoJEAF3yXsqtyBlM50P/iHt5rwT/sGWSaNnsBNNoS0L
WKb8Z9M7nVwaWsjceHPwEWMDrW2M7TUlXWCWhmDOL5oWP2PtX5J0YkXZ3ADmn5cp
GE8gKmFDIdoepxs/GQREpryh+mT+kyhr+3WIISgSplEsGP0PezEBEX5jemvMAFcn
bVH4KYj5Cqt/xludubqxaNe/GF72FwJKVl/ie5GIMF1gk039QpykOvI8GZzcXXVU
/vEUH72i+JOgyrLCMIuzH6na2YSiXI1pav8NnPV4pZCX6Rre8/MTtNGRd8Lda3zR
Nqb1Jow7ozTRwYWJpORU0ZiPN4aTQakktSLuPxN3KpAFOUiKbt4EMAI7YwceYh0b
DwE7fml1auINn2XhwLYyHyX6bu0TJQmC8PbOxtx2J79wO0707ZPpvNN2imgGqYbg
zdO7cQMlI04MqeRn9A+OgtLzAh/yrJaVDNYNN6OFbSpyfB0FrmrZpKxozX2gMOp2
C1WBffSflKN+RLAaWGsXY+CIDyHvkIJifUx+618O1iOqxXlWuMTmMa+Ez4DVpVLZ
SIoQBmcE950ZE6mZZazdb2rGVC1OhQVcvsIxv3qDqxPUkBP5rrbYhW3SziqaOZHj
M5o5iVkCRwBbxyV8GyVK8YYgBlu9CqxjMAfuNxT8aWZfl5kXVMxmfU8W2x4Pgjm1
n7ygIadGuw+cXEcr9ech
=8SBu
-END PGP SIGNATURE-
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Gordon Messmer]

On 11/13/2015 12:59 PM, J Martin Rushton wrote:

Maybe I should have been clearer: use (LVM) OR (RAID1 and break).


I took your meaning.  I'm saying that's a terrible backup strategy, for 
a list of reasons.


For instance, it only works if you mirror a single disk.  It doesn't 
work if you use RAID10 or RAID5, or RAID6, or RAIDZ, etc.  Breaking RAID 
doesn't make the data consistent, so you might have corrupt files 
(especially if the system runs any kind of database.  SQL, LDAP, etc).  
It doesn't make the filesystem consistent, so you might have a corrupt 
filesystem.


Even if you ignore the potential for corruption, you have a backup 
process that only works on some specific hardware configurations. 
Everything else has to have a different backup solution.  That's 
insane.  Use one backup process that works for everything.  You're much 
more likely to consistently back up your data that way.



I hope I'm wrong, but you wouldn't be thinking of mounting the broken
out copy on a the same system would you?  You must never do that, not
even during disaster recovery.  Use dd or similar on the disk, not the
mounted partitions - isn't that obvious?  I wasn't trying to give step
by step instructions.


Well, that's *one* of the problems with your advice.  Even if we ignore 
the fact that it doesn't work reliably (and IMO, it therefore doesn't 
work), it's far more complicated than you pretend it is.


Because now you're talking about quiescing your services, breaking your 
RAID, physically removing the drive, connecting it to another system, 
fsck the filesystems, mount them, and backing up the data. For each 
backup.  Every day.


Or using 'dd' and... backing up the whole image?  No incremental or 
differentials?


Your process involves a human being doing physical tasks as part of the 
backup.  Maybe I'm the only one, but I want my backups fully automated.  
People make mistakes.  I don't want them involved in regular processes.  
In fact, the entire point of computing is that the computer should do 
the work so that I don't have to.



Way before LVM existed we used this technique to back up VAXes (and
later Alphas) under VMS using "volume shadowing" (ie RAID1). It worked
quite happily for several years with disks shared across the cluster.
IIRC it was actually recommended by DEC, indeed a selling point, but I
don't have any manuals to hand to confirm that nowadays! One thing I
did omit was you MUST sync first


sync flushes the OS data buffers to disk, but it does not sync 
application data buffers, it does not flush the journal, it doesn't make 
filesystems "clean", and even if you break the RAID volume immediately 
after "sync" there's no guarantee that there weren't cached writes from 
other processes in between those two steps.


There is absolutely no way to make this a reliable process without a 
full shutdown.


___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Anthony K]

On 11/11/15 02:46, Gordon Messmer wrote:


... the process you described is likely to miss files that are 
modified while "find" runs.



That's just being picky for the sake of it.  A backup is a *point-in-time* 
snapshot of the files being backed up.  It will not capture files modified 
after that point.

So, saying that find won't find files modified while the backup is running is 
frankly the same as saying it won't find files modified anytime in the future 
after that *point-in-time* when the backup started!

If there's a point to be made by the quoted statement above, I missed it and I 
surely deserve to be educated!


ak.


___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Benjamin Smith]

I did exactly this with ZFS on Linux and cut over 24 hours of backup lag to 
just minutes. 

If you're managing data at scale, ZFS just rocks... 


On Tuesday, November 10, 2015 01:16:28 PM Warren Young wrote:
> On Nov 10, 2015, at 8:46 AM, Gordon Messmer  
wrote:
> > On 11/09/2015 09:22 PM, Arun Khan wrote:
> >> You can use "newer" options of the find command and pass the file list
> > 
> > the process you described is likely to miss files that are modified while
> > "find" runs.
> Well, be fair, rsync can also miss files if files are changing while the
> backup occurs.  Once rsync has passed through a given section of the tree,
> it will not see any subsequent changes.
> 
> If you need guaranteed-complete filesystem-level snapshots, you need to be
> using something at the kernel level that can atomically collect the set of
> modified blocks/files, rather than something that crawls the tree in user
> space.
> 
> On the BSD Now podcast, they recently told a war story about moving one of
> the main FreeBSD servers to a new data center.  rsync was taking 21 hours
> in back-to-back runs purely due to the amount of files on that server,
> which gave plenty of time for files to change since the last run.
> 
> Solution?  ZFS send:
> 
>   http://128bitstudios.com/2010/07/23/fun-with-zfs-send-and-receive/
> ___
> CentOS mailing list
> CentOS@centos.org
> https://lists.centos.org/mailman/listinfo/centos

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Gordon Messmer]

On 11/10/2015 11:27 PM, Arun Khan wrote:

rsync will do incremental backup as already discussed earlier in this thread.

Please suggest  how to achieve a differential backup with rsync (the
original query).


Already answered.  Under rsync based backup systems like rsnapshot, 
every backup is a full backup.  Therefore, incremental and differential 
backups are the same thing.  As you already understand that rsync will 
do incremental backups without using find, you also understand that it 
will do differential backups without using find.

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Gordon Messmer]

On 11/09/2015 09:22 PM, Arun Khan wrote:

You can use "newer" options of the find command and pass the file list
to rsync or scp to "backup" only those files that have changed since
the last run.  You can keep a file like .lastbackup and timestamp it
(touch) at the start of the backup process.  Next backup you compare
the current timestamp with the timestamp on this file.


Absolutely none of that is necessary with rsync, and the process you 
described is likely to miss files that are modified while "find" runs.


If you're going to use rsync to make backups, just use a frontend like 
rsnapshot or backuppc.

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Gordon Messmer]

On 11/10/2015 12:16 PM, Warren Young wrote:


Well, be fair, rsync can also miss files if files are changing while the backup 
occurs.  Once rsync has passed through a given section of the tree, it will not 
see any subsequent changes.


I think you miss my meaning.  Consider this sequence of events:

* "find" begins and processes dirA and then dirB
* another application writes files in dirA
* "find" completes
* a new timestamp file is written

Now, the new file in dirA wasn't seen by find during this run, and it 
won't be seen on the next run either.  That's what I mean by missed.  
Not temporarily missed, but permanently.  That file won't ever be backed 
up in this very naïve process.


There's no benefit to the process, either.  rsync can efficiently 
examine and synchronize filesystems without using find.  And while it 
may miss files that are written while it's running, it *will* get them 
on the next run, unlike using "find".



If you need guaranteed-complete filesystem-level snapshots, you need to be 
using something at the kernel level that can atomically collect the set of 
modified blocks/files, rather than something that crawls the tree in user space.


Generally, I agree with you.  In fact:
https://bitbucket.org/gordonmessmer/dragonsdawn-snapshot
https://github.com/rsnapshot/rsnapshot/pull/44

Doing block-level differentials is nice, if you're using ZFS.  But not 
everyone wants to run ZFS on Linux.  I do think that backing up 
snapshots is important, though.


___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Warren Young]

On Nov 10, 2015, at 8:46 AM, Gordon Messmer  wrote:
> 
> On 11/09/2015 09:22 PM, Arun Khan wrote:
>> You can use "newer" options of the find command and pass the file list
> 
> the process you described is likely to miss files that are modified while 
> "find" runs.

Well, be fair, rsync can also miss files if files are changing while the backup 
occurs.  Once rsync has passed through a given section of the tree, it will not 
see any subsequent changes.

If you need guaranteed-complete filesystem-level snapshots, you need to be 
using something at the kernel level that can atomically collect the set of 
modified blocks/files, rather than something that crawls the tree in user space.

On the BSD Now podcast, they recently told a war story about moving one of the 
main FreeBSD servers to a new data center.  rsync was taking 21 hours in 
back-to-back runs purely due to the amount of files on that server, which gave 
plenty of time for files to change since the last run. 

Solution?  ZFS send:

  http://128bitstudios.com/2010/07/23/fun-with-zfs-send-and-receive/
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[J Martin Rushton]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 10/11/15 21:05, Gordon Messmer wrote:
> On 11/10/2015 12:16 PM, Warren Young wrote:
>> 
>> Well, be fair, rsync can also miss files if files are changing
>> while the backup occurs.  Once rsync has passed through a given
>> section of the tree, it will not see any subsequent changes.
> 
> I think you miss my meaning.  Consider this sequence of events:
> 
> * "find" begins and processes dirA and then dirB * another
> application writes files in dirA * "find" completes * a new
> timestamp file is written
> 
> Now, the new file in dirA wasn't seen by find during this run, and
> it won't be seen on the next run either.  That's what I mean by
> missed. Not temporarily missed, but permanently.  That file won't
> ever be backed up in this very naïve process.

That's plain bad system analysis.  Read the start date, record the
current date and THEN start processing.  You will get the odd extra
file but will not loose any.

> 
> There's no benefit to the process, either.  rsync can efficiently 
> examine and synchronize filesystems without using find.  And while
> it may miss files that are written while it's running, it *will*
> get them on the next run, unlike using "find".
> 
>> If you need guaranteed-complete filesystem-level snapshots, you
>> need to be using something at the kernel level that can
>> atomically collect the set of modified blocks/files, rather than
>> something that crawls the tree in user space.
> 
> Generally, I agree with you.  In fact: 
> https://bitbucket.org/gordonmessmer/dragonsdawn-snapshot 
> https://github.com/rsnapshot/rsnapshot/pull/44
> 
> Doing block-level differentials is nice, if you're using ZFS.  But
> not everyone wants to run ZFS on Linux.  I do think that backing
> up snapshots is important, though.
> 
> ___ CentOS mailing
> list CentOS@centos.org 
> https://lists.centos.org/mailman/listinfo/centos
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iQIcBAEBAgAGBQJWQn/cAAoJEAF3yXsqtyBl6YwQAKRneHW0APPxY9znjXFcSiuC
M3r5a5cYDOIdXNkFRg3Psoa38vaiHnmV3BFWfWWZzo4qAYfUP9fNP5i9McXWslK+
Jv2S0s4WoUcvyguRStpw++HmOBJ4Hywr2qXQnnPNuzuJXLD/EgxGgA5hv0lbYAN1
fzci8UuGpZI/WsCMIib3f8u6tQ4CTsaZm7AUjAZ/EdxyhNl17bjAC/U9tWYpCKtr
DF5jEyU18ZiHcWDym45SI1fjaiRf8jR0OrennViFyThYVLsaYgRiHn0gZAl9H2jx
5oL0fONkNY0f4hNN0XFyJkbDfdF4QCNu6WBHByv5I2MlOBIZBOI5zWagRTZ1uLCG
mMvpnj2Mmbugq+8p+XAhyB/xvMLaIA5suUSmIDej+rylH5FgrSUa+aHflUNBWZcK
P7odBqEDIc27CCSckdrAxY16+Drs5gjfw0EFmKXVaZ2SIyeJ4gTKGororEdOH9bI
ry6ItSuKD732OLx33/j15YuEUBhzwGHYiuE4xCZCegKK64/zFvVtLVO6z6kBN17T
BtOMX6VOyfJl3zxmlMfiZx6A01TxSqOw+QS9Tepcz0tAauFGN3UOow/Kv2RnrwhZ
LwbjRqPto/0599owBromt6lzZSoI9/4JX6DS+XQeLOPsuq8z6myxZc6AXFxE20NR
G9NoAxU24PbTq4VtLiKV
=+3+X
-END PGP SIGNATURE-
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Benjamin Smith]

On Monday, November 09, 2015 09:50:52 AM Gordon Messmer wrote:
> > How I can perform a diff backup?
> 
> Save yourself a lot of trouble and use a front-end like rsnapshot or
> backuppc.

If I may, I'd like to put in a plug for ZFS: 

Combining rsync and ZFS, you can rsync, then make a ZFS snapshot, which gives 
you the best of both worlds: 

1) No messy filesystem with multiple directories full of hardlinks to manage. 
2) Immutable backups. 
3) Crazy efficient storage space, including built-in compression. Much more 
efficient than rsync + hard links.  
4) Ability to send the entire filesystem (binary perfect) to another system. 
5) Ability to upgrade and add storage space without taking it offline. 
6) Ability to "restore" a snapshot to read/write status in seconds with a 
clone that you can throw away later just as easily. 
7) Or you can skip rsync, do the snapshots on the source server, and replicate 
the snapshots with send/receive. 
8) Uses inexpensive, commodity hardware.

... and on and on 

We've moved *all* our backups to ZFS, the benefits are just too many. I'd like 
to plug BTRFS in a similar vein, but it's "not yet production ready" and it's 
been that way for a long, long time... 

Ben S
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Gordon Messmer]

On 11/10/2015 03:38 PM, J Martin Rushton wrote:

That's plain bad system analysis.  Read the start date, record the
current date and THEN start processing.  You will get the odd extra
file but will not loose any.


That's my point.  "find" doesn't do that and naïve implementations of 
the original suggestion are likely to do work poorly.  For no reason.  
Just don't use "find" to feed rsync a list of files to sync.  It's not 
more efficient, it might miss files, it won't sync deleted files, etc 
etc.  rsync is designed to synchronize two directory trees.  It doesn't 
need external helpers (except for a pipe, like ssh).

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[John Logsdon]

Folks

I have been using rsnapshot for years now. The only problem I've found is
that it is possible to run out of inodes. So my heads-up is that when you
create the file system, ensure you have more than the default inodes - I
usually multiply the default by 10. Otherwise you can find your 1Tb USB
drive failing after 259Mb and you can't then recover the files. Rather
embarrassing.

> On Mon, Nov 9, 2015 at 9:31 PM, Alessandro Baggi
>  wrote:
>> Hi list,
>> how to perform a differential backup using rsync?
>>
>> On web there is a great confusion about diff backup concept when
>> searched
>> with rsync.
>>
>> Users says diff because it copy only differences. For me differential is
>> backup from last full backup.
>>
>
> You can use "newer" options of the find command and pass the file list
> to rsync or scp to "backup" only those files that have changed since
> the last run.  You can keep a file like .lastbackup and timestamp it
> (touch) at the start of the backup process.  Next backup you compare
> the current timestamp with the timestamp on this file.
>
> HTH,
> -- Arun Khan
> ___
> CentOS mailing list
> CentOS@centos.org
> https://lists.centos.org/mailman/listinfo/centos
>


Best wishes

John

John Logsdon
Quantex Research Ltd
+44 161 445 4951/+44 7717758675

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[John R Pierce]

On 11/10/2015 12:18 AM, John Logsdon wrote:

I have been using rsnapshot for years now. The only problem I've found is
that it is possible to run out of inodes. So my heads-up is that when you
create the file system, ensure you have more than the default inodes - I
usually multiply the default by 10. Otherwise you can find your 1Tb USB
drive failing after 259Mb and you can't then recover the files. Rather
embarrassing.


or use a file system, like xfs, that has no static allocations.

--
john r pierce, recycling bits in santa cruz

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Arun Khan]

On Tue, Nov 10, 2015 at 10:52 AM, Arun Khan  wrote:
> On Mon, Nov 9, 2015 at 9:31 PM, Alessandro Baggi
>  wrote:
>> Hi list,
>> how to perform a differential backup using rsync?
>>
>> On web there is a great confusion about diff backup concept when searched
>> with rsync.
>>
>> Users says diff because it copy only differences. For me differential is
>> backup from last full backup.
>>
>
> You can use "newer" options of the find command and pass the file list
> to rsync or scp to "backup" only those files that have changed since
> the last run.  You can keep a file like .lastbackup and timestamp it
> (touch) at the start of the backup process.  Next backup you compare
> the current timestamp with the timestamp on this file.
>

Clarification -- for diffrential back ups, you should touch the file
only when you do the *full* backup.

-- Arun Khan
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[John Logsdon]

Thanks John - I haven't used XFS.

This issue arose on ext3 I think some years ago on a rather elderly
system.  If XFS avoids this that's great but if someone is still using
legacy systems, they need to be warned!


> On 11/10/2015 12:18 AM, John Logsdon wrote:
>> I have been using rsnapshot for years now. The only problem I've found
>> is
>> that it is possible to run out of inodes. So my heads-up is that when
>> you
>> create the file system, ensure you have more than the default inodes - I
>> usually multiply the default by 10. Otherwise you can find your 1Tb USB
>> drive failing after 259Mb and you can't then recover the files. Rather
>> embarrassing.
>
> or use a file system, like xfs, that has no static allocations.
>


Best wishes

John

John Logsdon
Quantex Research Ltd
+44 161 445 4951/+44 7717758675

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Arun Khan]

On Wed, Nov 11, 2015 at 5:39 AM, Gordon Messmer
 wrote:
> On 11/10/2015 03:38 PM, J Martin Rushton wrote:
>>
>> That's plain bad system analysis.  Read the start date, record the
>> current date and THEN start processing.  You will get the odd extra
>> file but will not loose any.
>
>
> That's my point.  "find" doesn't do that and naïve implementations of the
> original suggestion are likely to do work poorly.

< snip ...>

A good systems analysis is a must in whatever one does.  Be it system
admin, software developer, accountant, lawyer etc.

My suggestion about using "find" was in response to OP's
question/clarification on incremental/differential backup and I
assumed due diligence with respective to designing the script.


how to perform a differential backup using rsync?

On web there is a great confusion about diff backup concept when
searched with rsync.


rsync will do incremental backup as already discussed earlier in this thread.

Please suggest  how to achieve a differential backup with rsync (the
original query).

Thanks,
-- Arun Khan
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Arun Khan]

On Wed, Nov 11, 2015 at 5:08 AM, J Martin Rushton
 wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> On 10/11/15 21:05, Gordon Messmer wrote:
>> On 11/10/2015 12:16 PM, Warren Young wrote:
>>>
>>> Well, be fair, rsync can also miss files if files are changing
>>> while the backup occurs.  Once rsync has passed through a given
>>> section of the tree, it will not see any subsequent changes.
>>
>> I think you miss my meaning.  Consider this sequence of events:
>>
>> * "find" begins and processes dirA and then dirB * another
>> application writes files in dirA * "find" completes * a new
>> timestamp file is written
>>
>> Now, the new file in dirA wasn't seen by find during this run, and
>> it won't be seen on the next run either.  That's what I mean by
>> missed. Not temporarily missed, but permanently.  That file won't
>> ever be backed up in this very naïve process.
>
> That's plain bad system analysis.  Read the start date, record the
> current date and THEN start processing.  You will get the odd extra
> file but will not loose any.
>

Heartily agree.  I was about to post my response but saw yours.

Cheers,
-- Arun Khan
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Ole Holm Nielsen]

Alessandro Baggi  wrote:

how to perform a differential backup using rsync?

On web there is a great confusion about diff backup concept when
searched with rsync.


I think the answer to this question is Rsnapshot, which is an old and 
well proven tool: http://rsnapshot.org/.  To quote the homepage:

rsnapshot is a filesystem snapshot utility based on rsync. rsnapshot makes it 
easy to make periodic snapshots of local machines, and remote machines over 
ssh. The code makes extensive use of hard links whenever possible, to greatly 
reduce the disk space required.


It's very easy to set up Rsnapshot with a simple configuration file. You 
can define hourly, daily, weekly and monthly backups.  You run Rsnapshot 
from crontab and it's all automatic.


We run Rsnapshot on the backup server which NFS-mounts the source file 
system as read-only (NFS read operations are low overhead).  You may 
also use SSH.


The backup server has XFS filesystems with the inode64 mount option (no 
inode problems :-). Our backup filesystems varies from 1-40 TB in size 
with 100k to 25M inodes. For very large source filesystems you may want 
to use 10 Gbit/s links to speed up the backups.


The only issue I see is with source filesystems containing tens of 
millions of files, since new inodes have to be allocated on the backup 
server to "snapshot" a filesystem.  The time for Rsnapshot to do 
"snapshots" can become substantial, depending on the speed of the 
underlying filesystem.


--
Ole Holm Nielsen
Department of Physics, Technical University of Denmark,
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Frank Thommen]

Ciao Alessandro,

On 11/09/2015 05:01 PM, Alessandro Baggi wrote:

Hi list,
how to perform a differential backup using rsync?

On web there is a great confusion about diff backup concept when
searched with rsync.

Users says diff because it copy only differences. For me differential is
backup from last full backup.


Which is basically the same...if you always use your last full backup as 
"base" directory.  Use rsyn's --link-dest option to achieve this.  Nice 
thing: Unchanged files will just be hardlinked to the original files and 
won't use additional disk space, but still each dataset is a coopmlete 
backup.  There is no need to combine several incremental or differential 
backups to restore a certain state.


Mike Rubel's page has already been mentioned.  On 
http://www.drosera.ch/frank/computer/rsync.html I describe an alternate 
mechanism (using above mentioned --link-dest and an rsync-server) which 
overcomes some of the - imho - shortcomings of Mike's setup.


And: rsync is a fan-tas-tic backup tool ;-)

HTH
Frank


___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Valeri Galtsev]

On Mon, November 9, 2015 7:52 pm, Keith Keller wrote:
> On 2015-11-09, John R Pierce  wrote:
>>
>> XFS handles this fine.  I have a backuppc storage pool with backups of
>> 27 servers going back a year...  now, I just have 30 days of
>> incrementals, and 12 months of fulls,
>
> I'm sure you know this already, but for those who may not, be sure to
> mount your XFS filesystem with the inode64 option.  Otherwise XFS will
> try to save all of its inodes in the first 1TB of space, and with so
> many inodes needed, you may run out more quickly than you anticipate.
> Then you'll have "no space left on device" errors when df reports plenty
> of space (at least till you do df -i; actually I'm not 100% sure df -i
> will show it).

I'm fully with you on -o inode64, but I would think it is not inode number
that becomes large with extensive use of hard links, but the space used by
directory data, thus requiring to relocate these once they exceed some
size so ultimately some of them will be pushed beyond 1 TB border
(depending on how the filesystem is used). Someone, correct me if I'm
wrong.

Valeri

>
> --keith
>
> --
> kkel...@wombat.san-francisco.ca.us
>
>
> ___
> CentOS mailing list
> CentOS@centos.org
> https://lists.centos.org/mailman/listinfo/centos
>



Valeri Galtsev
Sr System Administrator
Department of Astronomy and Astrophysics
Kavli Institute for Cosmological Physics
University of Chicago
Phone: 773-702-4247

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Arun Khan]

On Mon, Nov 9, 2015 at 9:31 PM, Alessandro Baggi
 wrote:
> Hi list,
> how to perform a differential backup using rsync?
>
> On web there is a great confusion about diff backup concept when searched
> with rsync.
>
> Users says diff because it copy only differences. For me differential is
> backup from last full backup.
>

You can use "newer" options of the find command and pass the file list
to rsync or scp to "backup" only those files that have changed since
the last run.  You can keep a file like .lastbackup and timestamp it
(touch) at the start of the backup process.  Next backup you compare
the current timestamp with the timestamp on this file.

HTH,
-- Arun Khan
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Keith Keller]

On 2015-11-10, Valeri Galtsev  wrote:
>
> I'm fully with you on -o inode64, but I would think it is not inode number
> that becomes large with extensive use of hard links, but the space used by
> directory data, thus requiring to relocate these once they exceed some
> size so ultimately some of them will be pushed beyond 1 TB border
> (depending on how the filesystem is used). Someone, correct me if I'm
> wrong.

Does this answer the question you're asking?  I think so but I'm not
sure.

http://www.xfs.org/index.php/XFS_FAQ#Q:_What_is_the_inode64_mount_option_for.3F

--keith

-- 
kkel...@wombat.san-francisco.ca.us


___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Gordon Messmer]

On 11/09/2015 09:59 AM, John R Pierce wrote:

On 11/9/2015 9:50 AM, Gordon Messmer wrote:

I don't see the distinction you're making.


a incremental backup copies everything since the last incremental
a differential copies everything since the last full.


I guess that makes sense, but in backup systems based on rsync and hard 
links (such as rsnapshot), *every* backup on the backup volume is a 
"full" backup, so incremental and differential are the same thing.



rsync is NOT a backup system, its just a incremental file copy


..which can be used as a component of a backup system, such as rsnapshot 
or backuppc.


___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[m . roth]

Gordon Messmer wrote:
> On 11/09/2015 09:59 AM, John R Pierce wrote:
>> On 11/9/2015 9:50 AM, Gordon Messmer wrote:
>>> I don't see the distinction you're making.
>>
>> a incremental backup copies everything since the last incremental
>> a differential copies everything since the last full.
>
> I guess that makes sense, but in backup systems based on rsync and hard
> links (such as rsnapshot), *every* backup on the backup volume is a
> "full" backup, so incremental and differential are the same thing.
>
>> rsync is NOT a backup system, its just a incremental file copy
>
> ..which can be used as a component of a backup system, such as rsnapshot
> or backuppc.

Actually, we use rsync for backups. We have a script that creates a new
daily directory... and uses hard links to previous dates. That way, it
looks like a full b/u... but you can go to a previous date to restore an
older version of the file (aka ACK! I saved that file full of garbage to
my Great American Novel filename! ).

And if you aren't familiar with hard links, which rsync happily creates,
they were certainly hard enough to wrap my head around, until I got it...
and really like them. Just note that they *must* be on one filesystem, as
opposed to symlinks, which can cross filesystems.

mark

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[John R Pierce]

On 11/9/2015 9:50 AM, Gordon Messmer wrote:


I don't see the distinction you're making.


a incremental backup copies everything since the last incremental

a differential copies everything since the last full.

rsync is NOT a backup system, its just a incremental file copy


with the full/incremental/differential approach, a restore to a given 
date would need to restore the last full, then the last differential, 
then any incrementals since that differential, for instance, if you do 
monthly full, weekly differential and daily incrementals.If you 
don't use differentials, then you'd have to restore every incremental 
since that last full, which in a monthly full, daily incremental 
scenario could be as many as 30 incrementals.



--
john r pierce, recycling bits in santa cruz

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Gordon Messmer]

On 11/09/2015 08:01 AM, Alessandro Baggi wrote:

how to perform a differential backup using rsync?


rsync backups are always incremental against the most recent backup 
(assuming you're copying to the same location).


Users says diff because it copy only differences. For me differential 
is backup from last full backup.


I don't see the distinction you're making.

rsync examines each file.  If you specify --delete, files that are in 
the destination but not the source will be removed.  Generally, files 
that match last-modified-time and size will not be copied, but flags 
like -c change the criteria for determining whether a file needs to be 
copied.  Files which do not match will be copied using an efficient 
algorithm to send the minimum amount of data (just the changes in the 
file) from the source to the destination.


Other users says that to perform a differential backup I must include 
in rsync command: --backup --backup-dir=/some/path but from manual 
page of rsync:


You probably only need to use --backup-dir on systems which don't have 
GNU cp.  On systems with GNU cp, differential backups normally do 
something like:


cp -a daily.0 daily.1
rsync -a --delete source/ daily.0/

Whereas with --backup-dir, you can use rsync to do both tasks in one 
command, but your directory layout is a little messier.



How I can perform a diff backup?


Save yourself a lot of trouble and use a front-end like rsnapshot or 
backuppc.


___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[David Both]

I beg to differ.

The rsync command is a fantastic backup system. It may not meet your 
needs, but it works really great to make different types of backups for 
me. I have a script I use (automate everything) to perform nightly 
backups with rsync. Using rsync with USB external hard drives works far 
better than any other backup system I have ever tried.


As for your other statements, they may be meaningful to you and that is 
OK, but to me are just so much irrelevant semantics. If one's backup 
system works, terminology and which commands used to achieve it are 
beside the point - it is a true backup system.



On 11/09/2015 12:59 PM, John R Pierce wrote:

On 11/9/2015 9:50 AM, Gordon Messmer wrote:


I don't see the distinction you're making.


a incremental backup copies everything since the last incremental

a differential copies everything since the last full.

rsync is NOT a backup system, its just a incremental file copy


with the full/incremental/differential approach, a restore to a given 
date would need to restore the last full, then the last differential, 
then any incrementals since that differential, for instance, if you do 
monthly full, weekly differential and daily incrementals.If you 
don't use differentials, then you'd have to restore every incremental 
since that last full, which in a monthly full, daily incremental 
scenario could be as many as 30 incrementals.




--


*
David P. Both, RHCE
Millennium Technology Consulting LLC
Raleigh, NC, USA
919-389-8678

db...@millennium-technology.com

www.millennium-technology.com
www.databook.bz - Home of the DataBook for Linux
DataBook is a Registered Trademark of David Both
*
This communication may be unlawfully collected and stored by the National
Security Agency (NSA) in secret. The parties to this email do not consent to the
retrieving or storing of this communication and any related metadata, as well as
printing, copying, re-transmitting, disseminating, or otherwise using it. If you
believe you have received this communication in error, please delete it
immediately.


___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Valeri Galtsev]

On Mon, November 9, 2015 10:01 am, Alessandro Baggi wrote:
> Hi list,
> how to perform a differential backup using rsync?

Differential comes from real backup systems. Rsync is much simpler IMHO,
"-b" backup flag only keeps older version or deleted file/directory with
extra "~" (or whatever you define) in its name. Making rsync behaving as
full blown backup system is too time consuming. Much less time consuming
will be to just to install some backup software. Backuppc I would
recommend for simple case like I understand yours is. Bacula will be my
choice when I need enterprise level system.

Just my $0.02.

Valeri

>
> On web there is a great confusion about diff backup concept when
> searched with rsync.
>
> Users says diff because it copy only differences. For me differential is
> backup from last full backup.
>
> Other users says that to perform a differential backup I must include in
> rsync command: --backup --backup-dir=/some/path but from manual page of
> rsync:
>
> #
> --backup-dir=DIR
>In combination with the --backup option, this tells rsync
> to store all backups in the specified directory on the receiving side.
> This can be used for incremental backups.  You can additionally specify
> a backup suffix
>using the --suffix option (otherwise the files backed up
> in the specified directory will keep their original filenames).
> 
> ###
>
>
> Then at this point, I can perform a full backup copying base dir after
> last incremental. I can performa an incremental backup saving change on
> a specified destdir (using --backup-dir).
>
> How I can perform a diff backup?
>
> I know that rsync check differences using "the base dir". This dir have
> "the same content" of backupped source. To make incremental, this base
> is used.
> Supposing that I've 500 GB data on source. Make/sync the base-dir of
> 500GB.
> Running a full backup (the result file must be a fullbackup.tar.gz), at
> the end of the process I get a base-dir of 500GB and a .tar.gz of +/-
> 500GB compressed. Is correct make full backup, performing first an
> incremental backup on the base-dir and then compress it on a .tar.gz? Or
> is better resync all source in alternative destdir?
>
>
> In this example I've spent the double space for a full and a base-dir.
> 500GB Source vs 1TB for base-dir and a full.tar.gz. There is a way to
> performs other operation (incr and diff) without using the base and save
> disk space?
>
>
> Thanks in advance.
> ___
> CentOS mailing list
> CentOS@centos.org
> https://lists.centos.org/mailman/listinfo/centos
>



Valeri Galtsev
Sr System Administrator
Department of Astronomy and Astrophysics
Kavli Institute for Cosmological Physics
University of Chicago
Phone: 773-702-4247

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[melkor.kp]

Hi

For backups with rsync a recommend you to follow the approach discussed on
this website.
It provides you everything for getting a full backup and then the
incremental ones (deltas) using rsync.
The only thing you need in order to do that is that the hosting filesystem
supports hard links,

http://www.mikerubel.org/computers/rsync_snapshots/

Cheers,
Roberto Nebot

2015-11-09 17:01 GMT+01:00 Alessandro Baggi :

> Hi list,
> how to perform a differential backup using rsync?
>
> On web there is a great confusion about diff backup concept when searched
> with rsync.
>
> Users says diff because it copy only differences. For me differential is
> backup from last full backup.
>
> Other users says that to perform a differential backup I must include in
> rsync command: --backup --backup-dir=/some/path but from manual page of
> rsync:
>
> #
> --backup-dir=DIR
>   In combination with the --backup option, this tells rsync to
> store all backups in the specified directory on the receiving side. This
> can be used for incremental backups.  You can additionally specify a backup
> suffix
>   using the --suffix option (otherwise the files backed up in
> the specified directory will keep their original filenames).
> 
> ###
>
>
> Then at this point, I can perform a full backup copying base dir after
> last incremental. I can performa an incremental backup saving change on a
> specified destdir (using --backup-dir).
>
> How I can perform a diff backup?
>
> I know that rsync check differences using "the base dir". This dir have
> "the same content" of backupped source. To make incremental, this base is
> used.
> Supposing that I've 500 GB data on source. Make/sync the base-dir of 500GB.
> Running a full backup (the result file must be a fullbackup.tar.gz), at
> the end of the process I get a base-dir of 500GB and a .tar.gz of +/- 500GB
> compressed. Is correct make full backup, performing first an incremental
> backup on the base-dir and then compress it on a .tar.gz? Or is better
> resync all source in alternative destdir?
>
>
> In this example I've spent the double space for a full and a base-dir.
> 500GB Source vs 1TB for base-dir and a full.tar.gz. There is a way to
> performs other operation (incr and diff) without using the base and save
> disk space?
>
>
> Thanks in advance.
> ___
> CentOS mailing list
> CentOS@centos.org
> https://lists.centos.org/mailman/listinfo/centos
>



-- 
It's a dangerous business, Frodo, going out your door. You step onto the
road, and if you don't keep your feet, there's no knowing where you might
be swept off to.

Bilbo Baggins.

http://vanishedintheshell.apsila.org

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

[CentOS] Rsync and differential Backups

[Alessandro Baggi]

Hi list,
how to perform a differential backup using rsync?

On web there is a great confusion about diff backup concept when 
searched with rsync.


Users says diff because it copy only differences. For me differential is 
backup from last full backup.


Other users says that to perform a differential backup I must include in 
rsync command: --backup --backup-dir=/some/path but from manual page of 
rsync:


#
--backup-dir=DIR
  In combination with the --backup option, this tells rsync 
to store all backups in the specified directory on the receiving side. 
This can be used for incremental backups.  You can additionally specify 
a backup suffix
  using the --suffix option (otherwise the files backed up 
in the specified directory will keep their original filenames).


###


Then at this point, I can perform a full backup copying base dir after 
last incremental. I can performa an incremental backup saving change on 
a specified destdir (using --backup-dir).


How I can perform a diff backup?

I know that rsync check differences using "the base dir". This dir have 
"the same content" of backupped source. To make incremental, this base 
is used.

Supposing that I've 500 GB data on source. Make/sync the base-dir of 500GB.
Running a full backup (the result file must be a fullbackup.tar.gz), at 
the end of the process I get a base-dir of 500GB and a .tar.gz of +/- 
500GB compressed. Is correct make full backup, performing first an 
incremental backup on the base-dir and then compress it on a .tar.gz? Or 
is better resync all source in alternative destdir?



In this example I've spent the double space for a full and a base-dir.
500GB Source vs 1TB for base-dir and a full.tar.gz. There is a way to 
performs other operation (incr and diff) without using the base and save 
disk space?



Thanks in advance.
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Valeri Galtsev]

On Mon, November 9, 2015 12:42 pm, m.r...@5-cent.us wrote:
> Gordon Messmer wrote:
>> On 11/09/2015 09:59 AM, John R Pierce wrote:
>>> On 11/9/2015 9:50 AM, Gordon Messmer wrote:
 I don't see the distinction you're making.
>>>
>>> a incremental backup copies everything since the last incremental
>>> a differential copies everything since the last full.
>>
>> I guess that makes sense, but in backup systems based on rsync and hard
>> links (such as rsnapshot), *every* backup on the backup volume is a
>> "full" backup, so incremental and differential are the same thing.
>>
>>> rsync is NOT a backup system, its just a incremental file copy
>>
>> ..which can be used as a component of a backup system, such as rsnapshot
>> or backuppc.
>
> Actually, we use rsync for backups. We have a script that creates a new
> daily directory... and uses hard links to previous dates. That way, it
> looks like a full b/u... but you can go to a previous date to restore an
> older version of the file (aka ACK! I saved that file full of garbage to
> my Great American Novel filename! ).

I wonder how filesystem behaves when almost every file has some 400 hard
links to it. (thinking in terms of a year worth of daily backups).

Valeri

>
> And if you aren't familiar with hard links, which rsync happily creates,
> they were certainly hard enough to wrap my head around, until I got it...
> and really like them. Just note that they *must* be on one filesystem, as
> opposed to symlinks, which can cross filesystems.
>
> mark
>
> ___
> CentOS mailing list
> CentOS@centos.org
> https://lists.centos.org/mailman/listinfo/centos
>



Valeri Galtsev
Sr System Administrator
Department of Astronomy and Astrophysics
Kavli Institute for Cosmological Physics
University of Chicago
Phone: 773-702-4247

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Gordon Messmer]

On 11/09/2015 11:10 AM, Frank Cox wrote:

>And if you aren't familiar with hard links, which rsync happily creates,
>they were certainly hard enough to wrap my head around, until I got it...

More than one filename for a particular file.  What's difficult about that?


I think the difficult part is that so many people don't understand that 
EVERY regular file is a hard link.  It doesn't mean "more than one" at 
all.  A hard link is the association between a directory entry 
(filename) and an inode in the filesystem.


___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[John R Pierce]

On 11/9/2015 11:34 AM, Valeri Galtsev wrote:

I wonder how filesystem behaves when almost every file has some 400 hard
links to it. (thinking in terms of a year worth of daily backups).



XFS handles this fine.  I have a backuppc storage pool with backups of 
27 servers going back a year...  now, I just have 30 days of 
incrementals, and 12 months of fulls, but in backuppc's implementation 
the distinction between incremental and full is quite blurred as both 
are fully deduped across the whole pool via use of hard links.


 * Pool is 5510.40GB comprising 9993293 files and 4369 directories (as
   of 11/9 02:08),
 * Pool hashing gives 3452 repeated files with longest chain 54,
 * Nightly cleanup removed 737 files of size 1.64GB (around 11/9 02:08),
 * Pool file system was recently at 35% (11/9 11:44), today's max is
   35% (11/9 01:00) and yesterday's max was 36%.

There are 27 hosts that have been backed up, for a total of:

 * 441 full backups of total size 71125.43GB (prior to pooling and
   compression),
 * 623 incr backups of total size 20775.88GB (prior to pooling and
   compression).


so 90+TB of backups take 5.5TB of actual space.

--
john r pierce, recycling bits in santa cruz

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Александр Кириллов]

cp -a daily.0 daily.1


cp -al daily.0 daily.1

All these can be combined with an rsyncd module to allow read only root 
access to a remote system excluding the dirs you don't normally want to 
be backed up like /proc, /var/lib/mysql, /var/lib/libvirt, ...


Oops... My provider email gateway has been blacklisted by anti spam 
vigilantes.


___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Gordon Messmer]

On 11/09/2015 11:34 AM, Valeri Galtsev wrote:

I wonder how filesystem behaves when almost every file has some 400 hard
links to it. (thinking in terms of a year worth of daily backups).


Why do you think that would be a problem?

Most inodes have one hard link.  When that link is removed, the link 
count in the inode is decremented (inodes are reference-counted, you can 
see their ref count in "ls -l" output).  When the link count reaches 0 
and no open file descriptors exist, the inode is removed.


Creating more hard links just increases the ref count.  That's it. It's 
not a weird special case.

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Frank Cox]

On Mon, 9 Nov 2015 11:36:18 -0800
Gordon Messmer wrote:

> I think the difficult part is that so many people don't understand that 
> EVERY regular file is a hard link.  It doesn't mean "more than one" at 
> all.  A hard link is the association between a directory entry 
> (filename) and an inode in the filesystem.

Now that you point that out, I agree.  I never thought about it that way before 
since I've always looked at a hard link as a link that you create after you 
create the initial file, though they become interchangeable after that.

But you're absolutely right and I've learned something today.  Thanks!

-- 
MELVILLE THEATRE ~ Real D 3D Digital Cinema ~ www.melvilletheatre.com
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Frank Cox]

On Mon, 9 Nov 2015 13:42:08 -0500
m.r...@5-cent.us wrote:

> And if you aren't familiar with hard links, which rsync happily creates,
> they were certainly hard enough to wrap my head around, until I got it...

More than one filename for a particular file.  What's difficult about that?

> and really like them. Just note that they *must* be on one filesystem, as
> opposed to symlinks, which can cross filesystems.

Obviously, since a hard link is part of the file and directory structure of the 
filesystem.

-- 
MELVILLE THEATRE ~ Real D 3D Digital Cinema ~ www.melvilletheatre.com
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[John R Pierce]

On 11/9/2015 12:02 PM, Frank Cox wrote:

Now that you point that out, I agree.  I never thought about it that way before 
since I've always looked at a hard link as a link that you create after you 
create the initial file, though they become interchangeable after that.


on Unix systems, the actual 'file' is known as an inode, and is 
identified by a inode number.   Directories are other files that contain 
indexed directory entries with filenames pointing to these inodes.


the tricky thing with hard links is, you have to walk the whole 
directory tree of a given file system to find every entry pointing to 
the same inode if you want to identify these links.


--
john r pierce, recycling bits in santa cruz

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Valeri Galtsev]

On Mon, November 9, 2015 1:41 pm, Gordon Messmer wrote:
> On 11/09/2015 11:34 AM, Valeri Galtsev wrote:
>> I wonder how filesystem behaves when almost every file has some 400 hard
>> links to it. (thinking in terms of a year worth of daily backups).
>
> Why do you think that would be a problem?

Probably not. You are not impacting something that has notably finite
count (like inode count on given fs). You just use a bit more disk space
for metadata which is nothing (space wise) compared to data (the files
themselves). Thanks!

>
> Most inodes have one hard link.  When that link is removed, the link
> count in the inode is decremented (inodes are reference-counted, you can
> see their ref count in "ls -l" output).  When the link count reaches 0
> and no open file descriptors exist, the inode is removed.
>
> Creating more hard links just increases the ref count.  That's it. It's
> not a weird special case.
> ___
> CentOS mailing list
> CentOS@centos.org
> https://lists.centos.org/mailman/listinfo/centos
>



Valeri Galtsev
Sr System Administrator
Department of Astronomy and Astrophysics
Kavli Institute for Cosmological Physics
University of Chicago
Phone: 773-702-4247

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[Keith Keller]

On 2015-11-09, John R Pierce  wrote:
>
> XFS handles this fine.  I have a backuppc storage pool with backups of 
> 27 servers going back a year...  now, I just have 30 days of 
> incrementals, and 12 months of fulls,

I'm sure you know this already, but for those who may not, be sure to
mount your XFS filesystem with the inode64 option.  Otherwise XFS will
try to save all of its inodes in the first 1TB of space, and with so
many inodes needed, you may run out more quickly than you anticipate.
Then you'll have "no space left on device" errors when df reports plenty
of space (at least till you do df -i; actually I'm not 100% sure df -i
will show it).

--keith

-- 
kkel...@wombat.san-francisco.ca.us


___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Rsync and differential Backups

[m . roth]

Valeri Galtsev wrote:
>
> On Mon, November 9, 2015 12:42 pm, m.r...@5-cent.us wrote:
>> Gordon Messmer wrote:
>>> On 11/09/2015 09:59 AM, John R Pierce wrote:
 On 11/9/2015 9:50 AM, Gordon Messmer wrote:
> I don't see the distinction you're making.

 a incremental backup copies everything since the last incremental
 a differential copies everything since the last full.
>>>
>>> I guess that makes sense, but in backup systems based on rsync and hard
>>> links (such as rsnapshot), *every* backup on the backup volume is a
>>> "full" backup, so incremental and differential are the same thing.
>>>
 rsync is NOT a backup system, its just a incremental file copy

>> Actually, we use rsync for backups. We have a script that creates a new
>> daily directory... and uses hard links to previous dates. That way, it
>> looks like a full b/u... but you can go to a previous date to restore an
>> older version of the file (aka ACK! I saved that file full of garbage to
>> my Great American Novel filename! ).
>
> I wonder how filesystem behaves when almost every file has some 400 hard
> links to it. (thinking in terms of a year worth of daily backups).

That, I can't answer - what we have is "disaster recovery", not "archive",
so we only keep them for no more than five weeks.

On the other hand... a reasonable approach would be for, over maybe two
months old, to keep the first of the month, and rm everything else for the
month.

   mark

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync question

[Leon Fauster]

Am 11.09.2015 um 05:49 schrieb Robert Moskowitz :
> 
> 
> On 09/10/2015 11:19 PM, Carl E. Hartung wrote:
>> On Thu, 10 Sep 2015 22:50:47 -0400
>> Robert Moskowitz wrote:
>> 
>>> 
>>> $ rsync -ah --stats --delete -p 613  root@192.168.129.2:/etc/dhcp/
>>> /home/rgm/data/htt/httnet/homebase/new/dhcp
>>> Unexpected remote arg: root@192.168.129.2:/etc/dhcp/
>>> rsync error: syntax or usage error (code 1) at main.c(1330)
>>> [sender=3.1.1]
>>> 
>>> So not there yet.  :(
>> >From 'man rsync':
>> 
>> -p, --perms This option causes the receiving rsync to set the destination
>> permissions to be the same as the source per- missions.
>> 
>> Try this:
>> 
>> rsync -ah --stats --delete
>> root@192.168.129.2:613:/etc/dhcp/ /home/rgm/data/htt/httnet/homebase/new/dhcp
> 
> SSh is not parsing the port the way http does, it seems:
> 
> $ rsync -ah --stats root@192.168.129.2:613:/etc/dhcp/ 
> /home/rgm/data/htt/httnet/homebase/new/dhcp
> ssh: connect to host 192.168.129.2 port 22: No route to host
> rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
> rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]
> 
> The reason why I change my SSH port is a simple way to keep port knocker 
> robots away.  Different hosts use different ports...
> 

rsync -e 'ssh -p 613' -ah ...

--
LF

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync question

[Robert Moskowitz]

On 09/11/2015 10:21 AM, C Linus Hicks wrote:

  On 09/10/15, Robert Moskowitz wrote:

--- Quoted text --
SSh is not parsing the port the way http does, it seems:

$ rsync -ah --stats root@192.168.129.2:613:/etc/dhcp/
/home/rgm/data/htt/httnet/homebase/new/dhcp
ssh: connect to host 192.168.129.2 port 22: No route to host
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]

The reason why I change my SSH port is a simple way to keep port knocker
robots away. Different hosts use different ports...

-- End quote 

Oh, right, so you either need to put that in your .ssh/config file or use -e 
'ssh -p 613' on the rsync command.

The config file should look like this:
Host 192.168.129.2
  Port 613


So we end up back needing the -e option or modifying the config file.

thanks for your time.  Really.


___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync question

[C Linus Hicks]

 On 09/10/15, Robert Moskowitz wrote:

--- Quoted text --
SSh is not parsing the port the way http does, it seems:

$ rsync -ah --stats root@192.168.129.2:613:/etc/dhcp/ 
/home/rgm/data/htt/httnet/homebase/new/dhcp
ssh: connect to host 192.168.129.2 port 22: No route to host
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]

The reason why I change my SSH port is a simple way to keep port knocker 
robots away. Different hosts use different ports...

-- End quote 

Oh, right, so you either need to put that in your .ssh/config file or use -e 
'ssh -p 613' on the rsync command.

The config file should look like this:
Host 192.168.129.2
 Port 613

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync question

[Bowie Bailey]

On 9/11/2015 10:34 AM, Robert Moskowitz wrote:



On 09/11/2015 10:21 AM, C Linus Hicks wrote:

  On 09/10/15, Robert Moskowitz wrote:

--- Quoted text --
SSh is not parsing the port the way http does, it seems:

$ rsync -ah --stats root@192.168.129.2:613:/etc/dhcp/
/home/rgm/data/htt/httnet/homebase/new/dhcp
ssh: connect to host 192.168.129.2 port 22: No route to host
rsync: connection unexpectedly closed (0 bytes received so far) 
[Receiver]

rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]

The reason why I change my SSH port is a simple way to keep port knocker
robots away. Different hosts use different ports...

-- End quote 

Oh, right, so you either need to put that in your .ssh/config file or 
use -e 'ssh -p 613' on the rsync command.


The config file should look like this:
Host 192.168.129.2
  Port 613


So we end up back needing the -e option or modifying the config file.

thanks for your time.  Really.


It's fairly common on technical lists like this for people to become 
fixated on minor problems or inefficiencies in a command or 
configuration when the actual issue is more complicated.  Try not to let 
it bother you too much.


As for your original question, I'm not sure why the files weren't copied 
as expected.  I ran your exact command with only the server names, port, 
and destination directory changed and all the files were copied.  You 
can try running the command without a destination and it should return a 
list of files found in the source directories.  If it doesn't list 
everything, there is some problem with how the source files are being 
specified or something preventing them from being read.  Selinux is 
always suspected in cases of strange permission problems.


--
Bowie
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync question

[Robert Moskowitz]

On 09/11/2015 11:06 AM, Bowie Bailey wrote:

On 9/11/2015 10:34 AM, Robert Moskowitz wrote:



On 09/11/2015 10:21 AM, C Linus Hicks wrote:

  On 09/10/15, Robert Moskowitz wrote:

--- Quoted text --
SSh is not parsing the port the way http does, it seems:

$ rsync -ah --stats root@192.168.129.2:613:/etc/dhcp/
/home/rgm/data/htt/httnet/homebase/new/dhcp
ssh: connect to host 192.168.129.2 port 22: No route to host
rsync: connection unexpectedly closed (0 bytes received so far) 
[Receiver]

rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]

The reason why I change my SSH port is a simple way to keep port 
knocker

robots away. Different hosts use different ports...

-- End quote 

Oh, right, so you either need to put that in your .ssh/config file 
or use -e 'ssh -p 613' on the rsync command.


The config file should look like this:
Host 192.168.129.2
  Port 613


So we end up back needing the -e option or modifying the config file.

thanks for your time.  Really.


It's fairly common on technical lists like this for people to become 
fixated on minor problems or inefficiencies in a command or 
configuration when the actual issue is more complicated.  Try not to 
let it bother you too much.


As for your original question, I'm not sure why the files weren't 
copied as expected.  I ran your exact command with only the server 
names, port, and destination directory changed and all the files were 
copied.  You can try running the command without a destination and it 
should return a list of files found in the source directories.  If it 
doesn't list everything, there is some problem with how the source 
files are being specified or something preventing them from being 
read.  Selinux is always suspected in cases of strange permission 
problems.


There is really something going on, perhaps with Selinux.  The files DID 
get copied, but I cannot get any tool to list them on my F22 target!  I 
copy back to someplace else on the C7 system, and there they all are.  
If I copy into a nonexistant folder on my F22 notebook, they all show in 
the new directory.  Freaky.  Same thing when I backed up my /etc/dhcpd/ 
directory.  Only dhcpd.conf shows in the file listing.  The others came 
across, but don't show.



___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync question

[Robert Moskowitz]

On 09/09/2015 08:17 PM, Carl E. Hartung wrote:

On Wed, 9 Sep 2015 05:51:38 -0700 (PDT)
Mark Milhollan wrote:


On Tue, 8 Sep 2015, Carl E. Hartung wrote:

On Tue, 8 Sep 2015 10:25:33 -0700 (PDT) Mark Milhollan wrote:

-e specifies the *local* transport command to use

What?! Straight from the documentation:

"   -e, --rsh=COMMAND   specify the remote shell to use"

If only one can properly interpret the meaning...  COMMAND is the
local command to run to obtain a remote (non-interactive) shell, so
it isn't that the remote shell program (invoked once the transport is
connected) is being specified, i.e., it is which `remsh' to use.  The
-p and -l provided thereby may be necessary, since the one is not the
default and the other isn't known to us to be the same as the local
user, which you glossed over as if one never has a need to specify.

Orthogonal to Robert's problem, the switch from default use of rsh to
ssh has made it a requirement for (good) automation to always supply
a -e to ensure the correct command is used to account for all
potential versions of rsync that may be used.


, and in this case it also specifies the remote port (613) and user
(root).  Granted one should probably use their ssh configuration to
do that but it isn't realy "wrong" (to be questioned) to do it via
options.

I didn't explicitly state that it was "wrong," just implied
(correctly) that it was unnecessary.

Potentially unnecessary.  Just because you might see putting the port
and user in the ssh config file as the right thing to do, and which I
also do whenever possible, doesn't mean Robert necessarily wants to
or can do so, and after all -e does exist.  Your questioning its use
as you did implied using it is wrong, to which I object.  Luckily you
decided to reply to the list quoting me so eventually Robert was
supplied with the clue you didn't provide, that it might be
pre-configured.


I just tried the following:

rsync -ah --stats "ssh -p613 -l root" 192.168.192.2:/root/samba.PDC/ 
/home/rgm/data/htt/httnet/homebase/new/root/


And it failed with:

Unexpected remote arg: 192.168.192.2:/root/samba.PDC/
rsync error: syntax or usage error (code 1) at main.c(1330) [sender=3.1.1]

I tried again with:

rsync -ah --stats -e "ssh -p613 -l root" 192.168.192.2:/root/samba.PDC/ 
/home/rgm/data/htt/httnet/homebase/new/root/


and it worked.  This is what I read from the manpage, that "-e" is 
needed for the ssh command.



___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync question

[Robert Moskowitz]

On 09/10/2015 11:19 PM, Carl E. Hartung wrote:

On Thu, 10 Sep 2015 22:50:47 -0400
Robert Moskowitz wrote:


That will use the default port on the remote host, you can override
that without specifying the -e if required using -p .


$ rsync -ah --stats --delete -p 613  root@192.168.129.2:/etc/dhcp/
/home/rgm/data/htt/httnet/homebase/new/dhcp
Unexpected remote arg: root@192.168.129.2:/etc/dhcp/
rsync error: syntax or usage error (code 1) at main.c(1330)
[sender=3.1.1]

So not there yet.  :(

>From 'man rsync':

-p, --perms This option causes the receiving rsync to set the destination
 permissions to be the same as the source per- missions.

Try this:

rsync -ah --stats --delete
root@192.168.129.2:613:/etc/dhcp/ /home/rgm/data/htt/httnet/homebase/new/dhcp


SSh is not parsing the port the way http does, it seems:

$ rsync -ah --stats root@192.168.129.2:613:/etc/dhcp/ 
/home/rgm/data/htt/httnet/homebase/new/dhcp

ssh: connect to host 192.168.129.2 port 22: No route to host
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]

The reason why I change my SSH port is a simple way to keep port knocker 
robots away.  Different hosts use different ports...



___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync question

[Carl E. Hartung]

On Thu, 10 Sep 2015 22:50:47 -0400
Robert Moskowitz wrote:

> > That will use the default port on the remote host, you can override
> > that without specifying the -e if required using -p .  
> 
> 
> $ rsync -ah --stats --delete -p 613  root@192.168.129.2:/etc/dhcp/ 
> /home/rgm/data/htt/httnet/homebase/new/dhcp
> Unexpected remote arg: root@192.168.129.2:/etc/dhcp/
> rsync error: syntax or usage error (code 1) at main.c(1330)
> [sender=3.1.1]
> 
> So not there yet.  :(

>From 'man rsync':

-p, --perms This option causes the receiving rsync to set the destination
permissions to be the same as the source per- missions.

Try this:

rsync -ah --stats --delete
root@192.168.129.2:613:/etc/dhcp/ /home/rgm/data/htt/httnet/homebase/new/dhcp

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync question

[Robert Moskowitz]

On 09/10/2015 02:31 PM, C Linus Hicks wrote:

  On 09/10/15, C Linus Hicks wrote:

  On 09/10/15, Robert Moskowitz wrote:
 Quoted text 

Try this:
rsync -ah --stats 192.168.192.2:/root/samba.PDC/
/home/rgm/data/htt/httnet/homebase/new/root/
- End Quote --

Sorry, I was too fast, not paying attention, you want this I believe:

rsync -ah --stats root@192.168.192.2:/root/samba.PDC/
/home/rgm/data/htt/httnet/homebase/new/root/

That will use the default port on the remote host, you can override that without 
specifying the -e if required using -p .



$ rsync -ah --stats --delete -p 613  root@192.168.129.2:/etc/dhcp/ 
/home/rgm/data/htt/httnet/homebase/new/dhcp

Unexpected remote arg: root@192.168.129.2:/etc/dhcp/
rsync error: syntax or usage error (code 1) at main.c(1330) [sender=3.1.1]

So not there yet.  :(


___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync question

[Robert Moskowitz]

Ah, I should have checked your second email!

On 09/10/2015 02:31 PM, C Linus Hicks wrote:

  On 09/10/15, C Linus Hicks wrote:

  On 09/10/15, Robert Moskowitz wrote:
 Quoted text 

Try this:
rsync -ah --stats 192.168.192.2:/root/samba.PDC/
/home/rgm/data/htt/httnet/homebase/new/root/
- End Quote --

Sorry, I was too fast, not paying attention, you want this I believe:

rsync -ah --stats root@192.168.192.2:/root/samba.PDC/
/home/rgm/data/htt/httnet/homebase/new/root/

That will use the default port on the remote host, you can override that without 
specifying the -e if required using -p .
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos



___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync question

[Robert Moskowitz]

On 09/10/2015 02:23 PM, C Linus Hicks wrote:

  On 09/10/15, Robert Moskowitz wrote:
 Quoted text 
I just tried the following:

rsync -ah --stats "ssh -p613 -l root" 192.168.192.2:/root/samba.PDC/
/home/rgm/data/htt/httnet/homebase/new/root/

And it failed with:

Unexpected remote arg: 192.168.192.2:/root/samba.PDC/
rsync error: syntax or usage error (code 1) at main.c(1330) [sender=3.1.1]

I tried again with:

rsync -ah --stats -e "ssh -p613 -l root" 192.168.192.2:/root/samba.PDC/
/home/rgm/data/htt/httnet/homebase/new/root/

and it worked. This is what I read from the manpage, that "-e" is
needed for the ssh command.

- End Quote --

Try this:
rsync -ah --stats 192.168.192.2:/root/samba.PDC/
/home/rgm/data/htt/httnet/homebase/new/root/


And how is it determined that the ssh login is to be as root, not my userid?


___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync question

[C Linus Hicks]

 On 09/10/15, Robert Moskowitz wrote:
 Quoted text 
I just tried the following:

rsync -ah --stats "ssh -p613 -l root" 192.168.192.2:/root/samba.PDC/ 
/home/rgm/data/htt/httnet/homebase/new/root/

And it failed with:

Unexpected remote arg: 192.168.192.2:/root/samba.PDC/
rsync error: syntax or usage error (code 1) at main.c(1330) [sender=3.1.1]

I tried again with:

rsync -ah --stats -e "ssh -p613 -l root" 192.168.192.2:/root/samba.PDC/ 
/home/rgm/data/htt/httnet/homebase/new/root/

and it worked. This is what I read from the manpage, that "-e" is 
needed for the ssh command.

- End Quote --

Try this:
rsync -ah --stats 192.168.192.2:/root/samba.PDC/ 
/home/rgm/data/htt/httnet/homebase/new/root/

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] rsync question

[C Linus Hicks]

 On 09/10/15, C Linus Hicks wrote:

 On 09/10/15, Robert Moskowitz wrote:
 Quoted text 

Try this:
rsync -ah --stats 192.168.192.2:/root/samba.PDC/ 
/home/rgm/data/htt/httnet/homebase/new/root/
- End Quote --

Sorry, I was too fast, not paying attention, you want this I believe:

rsync -ah --stats root@192.168.192.2:/root/samba.PDC/ 
/home/rgm/data/htt/httnet/homebase/new/root/

That will use the default port on the remote host, you can override that 
without specifying the -e if required using -p .
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos