Re: Incrementally backup ONLY deleted / changed / new files - SOLVED

[Philip Rhoades via rsync]

People,


On 2023-04-16 18:06, Philip Rhoades via rsync wrote:

People,

I have command for backing up to a backup dir (eg /home/backup/prix/) 
with:


  rsync -av --delete --link-dest /home/backup/prix/${1}_0_hard_link
/$1/ /home/backup/prix/${1}_$dt/

for example the etc dir - which creates the following in the backup 
dir:


  etc
  etc_0_hard_link
  etc_20230416_165232

- having previously created a hard-linked clone of the previously
existing backup eg:

  cp -al /home/backup/prix/etc /home/backup/prix/etc_hard_link

However, in the backed up dirs with the date-time extension, ALL the
files from etc are copied - how do I rsync ONLY the deleted / changed
/ new files?



After much Googling, I found this did what want:


#!/bin/bash

if test $# != 3
then echo "Usage: rsync_inc.sh full_path_source_dir 
full_path_comp_dest_dir full_path_diff_dir"

exit
fi

dt=`dt_Ymd_HMS.sh`

rsync -rcC --compare-dest=$2/ $1/ ${3}_$dt/


P.
--
Philip Rhoades

PO Box 896
Cowra  NSW  2794
Australia
E-mail:  p...@pricom.com.au

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Incrementally backup ONLY deleted / changed / new files

[Philip Rhoades via rsync]

People,

I have command for backing up to a backup dir (eg /home/backup/prix/) 
with:


  rsync -av --delete --link-dest /home/backup/prix/${1}_0_hard_link /$1/ 
/home/backup/prix/${1}_$dt/


for example the etc dir - which creates the following in the backup dir:

  etc
  etc_0_hard_link
  etc_20230416_165232

- having previously created a hard-linked clone of the previously 
existing backup eg:


  cp -al /home/backup/prix/etc /home/backup/prix/etc_hard_link

However, in the backed up dirs with the date-time extension, ALL the 
files from etc are copied - how do I rsync ONLY the deleted / changed / 
new files?


Thanks,

Phil.
--
Philip Rhoades

PO Box 896
Cowra  NSW  2794
Australia
E-mail:  p...@pricom.com.au

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Copying TBs -> error -> work around

[Philip Rhoades via rsync]

raf,


On 2020-09-14 10:07, raf via rsync wrote:

On Fri, Sep 11, 2020 at 10:53:14AM +1000, Philip Rhoades via rsync
 wrote:


Roland,


On 2020-09-10 21:27, Roland wrote:
> > with rsync hanging - after breakout on /home for writing I then get:
> > "Read-only file system"
>
> if your filesystem switches to read-only, you have a serious problem
> with your system/storage, not with rsync.
>
> rsync (or the workload) is simply triggering the problem.


Thanks for the response . .

Hmm . . but the drive that goes read-only is being read FROM not TO . 
. it

is hard to see how that should be an issue?
The backstory is that a relatively recent internal 8TB Seagate 
Barracuda had
its 7.2TB sda5 (home) partition corrupted - which itself was 
suspicious but
not impossible of course - so I had to switch temporarily to an 
external USB
4TB drive (which was a backup drive and was already up-to-date) for 
/home.

So now this exercise is rsyncing back to a NEW internal 8TB Seagate
Barracuda (sda5 again) . .

If you are correct about rsync simply triggering an existing problem 
on the
4TB USB drive, would that problem going to be recognised by a fsck 
(ext4)?

I will check this out after I switch over to the new internal sda5 for
/home.

Thanks,
Phil.


file systems can be remounted read only when there are
too many errors. perhaps that applies to read errors as
well, not just write errors. check logs for i/o errors.
if it were i/o errors that caused the kernel to remount
the file system read only, it should have logged those
errors. and you should be able to use fsck with a usb
drive.



Hmm . . well I gave up trying to rsync the nearly whole 4TB at once and 
broke it down into individual dirs like I described in the OP but after 
that I did actually look at the 4TB USB "from" drive and there wasn't 
much wrong with it:


# fsck.ext4 /dev/sdc -y
e2fsck 1.45.3 (14-Jul-2019)
/dev/sdc contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Inode 9860147 extent tree (at level 1) could be shorter.  Optimize? yes

Pass 1E: Optimizing extent trees
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/sdc: * FILE SYSTEM WAS MODIFIED *
/dev/sdc: 10583339/244195328 files (0.4% non-contiguous), 
806625360/976754645 blocks


So then, from the corrupted original /dev/sda5 I tried to create an 
image to a second, new 8TB drive with an ext4 partition on the whole 
drive:


# ddrescue /dev/sda5 -d -r3 /mnte/sda5_ddrescue.img 
/root/sda5_ddrescue.mapfile

GNU ddrescue 1.25
Press Ctrl-C to interrupt
 ipos:9253 MB, non-trimmed:0 B,  current rate:   0 
B/s
 opos:9253 MB, non-scraped:0 B,  average rate:  37768 
kB/s
non-tried:7849 GB,  bad-sector:0 B,error rate:   0 
B/s
  rescued:9253 MB,   bad areas:0,run time:  4m  
5s
pct rescued:0.11%, read errors:0,  remaining time: 14h 
47m
  time since last successful read: 
13s

Copying non-tried blocks... Pass 1 (forwards)
ddrescue: Write error: Read-only file system

So I am getting a FS changed to RO in _two different_ situations - I 
think there is some OS (or motherboard?) problem . . after I update from 
Fedora 31 to 32 on one of the new 8TB drives, I might go through the 
exercise again to see if the problem is still there . .


Thanks,

Phil.
--
Philip Rhoades

PO Box 896
Cowra  NSW  2794
Australia
E-mail:  p...@pricom.com.au

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Copying TBs -> error -> work around

[Philip Rhoades via rsync]

Roland,


On 2020-09-10 21:27, Roland wrote:

with rsync hanging - after breakout on /home for writing I then get:
"Read-only file system"


if your filesystem switches to read-only, you have a serious problem
with your system/storage, not with rsync.

rsync (or the workload) is simply triggering the problem.



Thanks for the response . .

Hmm . . but the drive that goes read-only is being read FROM not TO . . 
it is hard to see how that should be an issue?


The backstory is that a relatively recent internal 8TB Seagate Barracuda 
had its 7.2TB sda5 (home) partition corrupted - which itself was 
suspicious but not impossible of course - so I had to switch temporarily 
to an external USB 4TB drive (which was a backup drive and was already 
up-to-date) for /home.  So now this exercise is rsyncing back to a NEW 
internal 8TB Seagate Barracuda (sda5 again) . .


If you are correct about rsync simply triggering an existing problem on 
the 4TB USB drive, would that problem going to be recognised by a fsck 
(ext4)?  I will check this out after I switch over to the new internal 
sda5 for /home.


Thanks,

Phil.



regards
roland


Am 10.09.20 um 07:30 schrieb Philip Rhoades via rsync:

People,

When I did:

  rsync -av /home/ /mntb5/  # about 4TB

I got errors like:

  'rsync [sender] expand file_list pointer array to xxx bytes, "did
move"'

with rsync hanging - after breakout on /home for writing I then get:

  "Read-only file system"

So after unmounting and remounting /home I did:

  cd /home
  find /home/ -type d | sort > ./home_dirs_sorted.txt

delete first line "/home/" of ./home_dirs_sorted.txt then:

  while read dir ; do echo $dir ; rsync -lptgod "$dir" /mntb5/"$dir" ;
done < ./home_dirs_sorted.txt

and:

  while read dir ; do echo $dir ; rsync -lptgoD "$dir"/\.[a-zA-Z0-9]*
/mntb5/"$dir"/ ; done < ./home_dirs_sorted.txt

and finally with no problems:

  rsync -av --exclude-from=/usr/local/bin/nfb_caches.txt /home/ 
/mntb5/


If there was a more sensible / efficient way of getting this done I
would like to know about it!

Thanks,

Phil.



--
Philip Rhoades

PO Box 896
Cowra  NSW  2794
Australia
E-mail:  p...@pricom.com.au

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Copying TBs -> error -> work around

[Philip Rhoades via rsync]

People,

When I did:

  rsync -av /home/ /mntb5/  # about 4TB

I got errors like:

  'rsync [sender] expand file_list pointer array to xxx bytes, "did 
move"'


with rsync hanging - after breakout on /home for writing I then get:

  "Read-only file system"

So after unmounting and remounting /home I did:

  cd /home
  find /home/ -type d | sort > ./home_dirs_sorted.txt

delete first line "/home/" of ./home_dirs_sorted.txt then:

  while read dir ; do echo $dir ; rsync -lptgod "$dir" /mntb5/"$dir" ; 
done < ./home_dirs_sorted.txt


and:

  while read dir ; do echo $dir ; rsync -lptgoD "$dir"/\.[a-zA-Z0-9]* 
/mntb5/"$dir"/ ; done < ./home_dirs_sorted.txt


and finally with no problems:

  rsync -av --exclude-from=/usr/local/bin/nfb_caches.txt /home/ /mntb5/

If there was a more sensible / efficient way of getting this done I 
would like to know about it!


Thanks,

Phil.

--
Philip Rhoades

PO Box 896
Cowra  NSW  2794
Australia
E-mail:  p...@pricom.com.au

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: linux rsync <-> SSHDroid has started becoming unreliable after an upgrade of Fedora 28 to 29

[Philip Rhoades via rsync]

Paul,


On 2019-02-03 03:00, Paul Slootman via rsync wrote:

On Sun 03 Feb 2019, Philip Rhoades via rsync wrote:


For some years I have been using rsync quite happily to send / 
retrieve
files to / from SSHDroid Pro but recently I have started having a 
problem
when transferring large numbers of file - I am pretty sure it started 
after
upgrading from Fedora x86_64 28 to 29 - but I am not 100% sure.  Below 
is


[...]


Corrupted MAC on input.
ssh_dispatch_run_fatal: Connection to 192.168.1.100 port 22: message
authentication code incorrect


ssh's communication gets disrupted somehow, and stops the connection,
thus causing rsync to fail.

This is a problem with ssh, not with rsync. Try enabling ssh debug
options, and try using different ssh ciphers.



I tried putting all the Ciphers one-by-one in:

  ~/.ssh/config

but with no improvement.  I will try ssh debugging . .

I have also sent a note to the developer of SSHDroid to see if (s)he 
responds but the last update to that app was in 2015 and ssh doesn't 
change very much so it is hard to see why that should suddenly be 
playing up - I will also try and use the ssh from Fedora 27/8 to see if 
that makes a difference.


Thanks,

Phil.
--
Philip Rhoades

PO Box 896
Cowra  NSW  2794
Australia
E-mail:  p...@pricom.com.au

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

linux rsync <-> SSHDroid has started becoming unreliable after an upgrade of Fedora 28 to 29

[Philip Rhoades via rsync]

People,

For some years I have been using rsync quite happily to send / retrieve 
files to / from SSHDroid Pro but recently I have started having a 
problem when transferring large numbers of file - I am pretty sure it 
started after upgrading from Fedora x86_64 28 to 29 - but I am not 100% 
sure.  Below is the tail end of the output of:


  rsync -avvv root@192.168.1.100:/storage . > err.txt 2>&2

.
.
sending file_sum
got file_sum
set modtime of 
storage/emulated/0/Android/data/com.google.android.youtube/cache/exo/.19.60878.1538877264113.v3.exo.lj1voL 
to (1538877264) Sun Oct  7 12:54:24 2018
renaming 
storage/emulated/0/Android/data/com.google.android.youtube/cache/exo/.19.60878.1538877264113.v3.exo.lj1voL 
to 
storage/emulated/0/Android/data/com.google.android.youtube/cache/exo/19.60878.1538877264113.v3.exo

false_alarms=0 hash_hits=0 matches=0
sender finished 
//storage/emulated/0/Android/data/com.google.android.youtube/cache/exo/19.60878.1538877264113.v3.exo
send_files(2996, 
//storage/emulated/0/Android/data/com.google.android.youtube/cache/exo/19.818.1538877263732.v3.exo)
send_files mapped 
//storage/emulated/0/Android/data/com.google.android.youtube/cache/exo/19.818.1538877263732.v3.exo 
of size 60060

recv_files(storage/emulated/0/Android/data/com.google.android.youtube/cache/exo/19.818.1538877263732.v3.exo)
storage/emulated/0/Android/data/com.google.android.youtube/cache/exo/19.818.1538877263732.v3.exo
calling match_sums 
//storage/emulated/0/Android/data/com.google.android.youtube/cache/exo/19.818.1538877263732.v3.exo

sending file_sum
got file_sum
set modtime of 
storage/emulated/0/Android/data/com.google.android.youtube/cache/exo/.19.818.1538877263732.v3.exo.825nBl 
to (1538877263) Sun Oct  7 12:54:23 2018
renaming 
storage/emulated/0/Android/data/com.google.android.youtube/cache/exo/.19.818.1538877263732.v3.exo.825nBl 
to 
storage/emulated/0/Android/data/com.google.android.youtube/cache/exo/19.818.1538877263732.v3.exo

false_alarms=0 hash_hits=0 matches=0
sender finished 
//storage/emulated/0/Android/data/com.google.android.youtube/cache/exo/19.818.1538877263732.v3.exo
send_files(2997, 
//storage/emulated/0/Android/data/com.google.android.youtube/cache/exo/2.3339089.1529636238000.v3.exo)
send_files mapped 
//storage/emulated/0/Android/data/com.google.android.youtube/cache/exo/2.3339089.1529636238000.v3.exo 
of size 1655452

recv_files(storage/emulated/0/Android/data/com.google.android.youtube/cache/exo/2.3339089.1529636238000.v3.exo)
storage/emulated/0/Android/data/com.google.android.youtube/cache/exo/2.3339089.1529636238000.v3.exo
calling match_sums 
//storage/emulated/0/Android/data/com.google.android.youtube/cache/exo/2.3339089.1529636238000.v3.exo

Corrupted MAC on input.
ssh_dispatch_run_fatal: Connection to 192.168.1.100 port 22: message 
authentication code incorrect
rsync: connection unexpectedly closed (18049139 bytes received so far) 
[receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) 
[receiver=3.1.3]
[receiver] _exit_cleanup(code=12, file=io.c, line=226): about to call 
exit(12)
rsync: connection unexpectedly closed (146209 bytes received so far) 
[generator]

rsync error: unexplained error (code 255) at io.c(226) [generator=3.1.3]
[generator] _exit_cleanup(code=12, file=io.c, line=226): about to call 
exit(255)


Suggestions about how to further debug this problem would be greatly 
appreciated!


Regards,

Phil.
--
Philip Rhoades

PO Box 896
Cowra  NSW  2794
Australia
E-mail:  p...@pricom.com.au

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsyncing without RSH or SSH ?

[Philip Rhoades via rsync]

Brian,


On 2017-03-21 07:35, Brian K. White via rsync wrote:

Any possibility of a version of rsync that doesn't need RSH or SSH?


rsync already doesn't need rsh or ssh.

* On host A (server, with room to accept big uploads):

Edit /etc/rsyncd.conf, add this to the end:
[hostb]
path = /srv/rsync/hostb
auth users = hostb

Edit /etc/rsyncd.secrets, add this line:
hostb:hostbpass

mkdir -p /srv/rsync/hostb

sudo rsync --daemon


* On host B (client, uploading to server):

RSYNC_PASSWORD=hostbpass rsync -avz /path/to/dir hostb@hosta::hostb

Replace "hosta" with whatever hostname or ip address for "Host A".

On hosta, the rsync --daemon wil listen on port 873 and the traffic
will go direct from rsync on hostb to rsync on hosta, no rsh or ssh.



Excellent!  Thanks for that . . I love rsync . . it is so clever . .



For single huge files like dd images though, dd|netcat is a lot faster.



Right.

Regards,

Phil.
--
Philip Rhoades

PO Box 896
Cowra  NSW  2794
Australia
E-mail:  p...@pricom.com.au

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsyncing without RSH or SSH ?

[Philip Rhoades via rsync]

Kevin,

I know a KK - I have talked to him about Sydney Futurists stuff and 
politics etc . . different email address though . .


See inline comments:


On 2017-03-14 16:17, Kevin Korb via rsync wrote:

Rsh should be pretty easy to do.  I don't think it has changed since
before RH5.2 vintage.



Yes, I was thinking I could cross-compile for 486 on my workstations - I 
will check that option out . .




Otherwise, your best bet would be NFS.  I think RH5.2 is new enough
for NFSv3 but I don't remember for sure.



Ah . . yes - good thinking - I will have a look at that too.  In recent 
years I have been using sshfs on odd occasions and I have forgotten 
about NFS . .




Alternatively maybe you know someone with a newer SCSI controller.  I
have several PCI ones and even a PCI-X one I haven't used in years.



Yes, I do have a PCI card in a box somewhere too - that will be last 
resort if the software approach fails . .


I have also sent a note to the Aboriginal Linux list and started 
building from their git repo - all very interesting and fun when I 
should be doing else . .


Regards,

Phil.



On 03/14/2017 12:30 AM, Philip Rhoades via rsync wrote:

People,

This doesn't seem possible - would it be possible to hack a version of
rsync to do it?

As an exercise, I want to create a VM image (or at least backup of all
the files with rsync) from a SCSI drive in an old Red Hat (NOT
Enterprise) v5.2 Linux 486 machine (circa 1999 that does have rsync on
it but it has protocol problems talking my server).  The 486 has:

- an ISA Adaptec 1542 SCSI controller attached to a Seagate 2GB drive
with 6 partitions

- an ISA ne2000 network card

- a 1.44MB FD

So, to be able to create the image of the 2GB SCSI drive that is in 
the

486, I need to boot the 486 on one or more floppy disks and dd the
/dev/sda drive over SSH to a file on my big Fedora 25 x86_64
workstation.  I now have Basic Linux booting from floppies but it is
still 2005 vintage and uses SSH1 and I haven't been able to get the 
old
Ciphers, MACs and KexAlgorithms working with my Fedora 25 x86_64 
server

- so I still can't ssh or rsync to it from the 486.  BL does not have
any rshd . .

Any possibility of a version of rsync that doesn't need RSH or SSH?
Even if that hacking were possible, I guess it would have to be done 
on
the old rsync code for RH5.2 or BL3.5 . . and I don't know if that 
stuff

would still exist anywhere anyway . .

Thanks,

Phil.


--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,


--
Philip Rhoades

PO Box 896
Cowra  NSW  2794
Australia
E-mail:  p...@pricom.com.au

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Rsyncing without RSH or SSH ?

[Philip Rhoades via rsync]

People,

This doesn't seem possible - would it be possible to hack a version of 
rsync to do it?


As an exercise, I want to create a VM image (or at least backup of all 
the files with rsync) from a SCSI drive in an old Red Hat (NOT 
Enterprise) v5.2 Linux 486 machine (circa 1999 that does have rsync on 
it but it has protocol problems talking my server).  The 486 has:


- an ISA Adaptec 1542 SCSI controller attached to a Seagate 2GB drive 
with 6 partitions


- an ISA ne2000 network card

- a 1.44MB FD

So, to be able to create the image of the 2GB SCSI drive that is in the 
486, I need to boot the 486 on one or more floppy disks and dd the 
/dev/sda drive over SSH to a file on my big Fedora 25 x86_64 
workstation.  I now have Basic Linux booting from floppies but it is 
still 2005 vintage and uses SSH1 and I haven't been able to get the old 
Ciphers, MACs and KexAlgorithms working with my Fedora 25 x86_64 server 
- so I still can't ssh or rsync to it from the 486.  BL does not have 
any rshd . .


Any possibility of a version of rsync that doesn't need RSH or SSH?  
Even if that hacking were possible, I guess it would have to be done on 
the old rsync code for RH5.2 or BL3.5 . . and I don't know if that stuff 
would still exist anywhere anyway . .


Thanks,

Phil.
--
Philip Rhoades

PO Box 896
Cowra  NSW  2794
Australia
E-mail:  p...@pricom.com.au

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html