Re: rsync to NAS for backup
On Mon, Feb 15, 2021 at 12:39:29PM +, mick crane wrote: > Appears that to retain permissions need root at both ends of rsync. Not necessarily. If the server filesystem supports xattrs, you can use the --fake-super option with the rsync server, running as a non-root user that can write to the storage. To set the option on the client side for rsync over ssh, use: --rsync-path "rsync --fake-super" Regards, Mirko
Re: rsync to NAS for backup
On Thu, Feb 18, 2021 at 05:06:33PM -0700, Charles Curley wrote: > On Thu, 18 Feb 2021 18:24:43 +0100 > wrote: > > > Care to name some of those limitations? > > > > (of course, rsync /is not/ a backup program in itself, but lets you > > build one with 10-30 lines around it). > > Why build one (which I have done) when you can get a perfectly good one > from the Debian repos? E.g.: rsnapshot. Others have been mentioned on > this thread. My approach is: if reading the doc takes me significantly longer than building, I do the latter. In this case, this is... uh... the case :-) YMMV, TANSTAAFL and all that. For me, this is the end of the thread (I feel we've been everywhere, everyone's happy with her solution and that). Cheers > -- > Does anybody read signatures any more? I do, yes. - t signature.asc Description: Digital signature
Re: rsync to NAS for backup
On Thu, 18 Feb 2021 18:24:43 +0100 wrote: > Care to name some of those limitations? > > (of course, rsync /is not/ a backup program in itself, but lets you > build one with 10-30 lines around it). Why build one (which I have done) when you can get a perfectly good one from the Debian repos? E.g.: rsnapshot. Others have been mentioned on this thread. -- Does anybody read signatures any more? https://charlescurley.com https://charlescurley.com/blog/ pgpnL4mGr3L7u.pgp Description: OpenPGP digital signature
Re: rsync to NAS for backup
On 2021-02-18 12:22, to...@tuxteam.de wrote: On Thu, Feb 18, 2021 at 06:59:03PM +0200, Teemu Likonen wrote: * 2021-02-18 11:13:25-0500, Gary Dale wrote: rsync is a quick & dirty backup tactic but it's got limitations. 1) files may stay around forever in the backup even if you've deleted them from your main computer because you don't need them. 2) you only have one copy of a file and that only lasts until the next rsync. This limits your ability to restore from a backup before it is overwritten. rsync is not a good substitute for backups. No, it's not. It is a fantastic tool for backups :-) Rsync is great backup program with "--link-dest" option. Here is the idea in simplified code: [...] Absolutely. Time travel! Actually, I've implemented this at a customer's place. They were delighted. Where rsync shows some weaknesses is on big, fat files (think videos, one or several GB). Really huge directories (tens to hundreds of TB) were once a challenge, too, but I hear that they refined the scanning part in the meantime. No direct experience, though. And, oh, Gary: if you want to delete files which disappeared in the source, check out the --delete option. But this time-staggered backup with --link-dest is really great. Cheers While you can twist any tool to fit a task, real backup programs don't need to be twisted and do a better job. For example backup retention policy is intuitive and easy to set. Some backup programs even factor out common blocks for de-duplication, which can save a lot of space. Hard-links only do that if the file name is the same. And when you need to restore a file, backup programs usually let you see when the files changed then let you choose which version to restore. As for the delete option, it makes the rsync script even more complicated. A backup program will simply expire the file at the end of the retention period.
Re: rsync to NAS for backup
On 2021-02-18 10:57, mick crane wrote: On 2021-02-15 12:39, mick crane wrote: On 2021-02-13 19:20, David Christensen wrote: On 2021-02-13 01:27, mick crane wrote: I made a mistake and instead of getting a PC for backup I got a NAS. I'm struggling to get to grips with it. If rsync from PC to NAS NAS changes the owner/group of files to me/users which is probably no good for backing up. There's that problem then another that it won't let me login as root. I asked on Synology forum but not getting a lot of joy. https://community.synology.com/enu/forum/1/post/141137 Anybody used these things can advise ? What is the model of the Synology NAS? What options -- CPU, memory, disks, bays, interfaces, PSU, whatever? Support page URL? Reading the forum post, it sounds like you damaged the sudoers file. The fix would appear to be doing a Mode 2 reset per Synology's instructions: https://www.synology.com/en-global/knowledgebase/DSM/tutorial/General_Setup/How_to_reset_my_Synology_NAS Once the NAS has been reset, figure out how to meet your needs within the framework provided by Synology. Follow the User Guide. Follow the Admin Guide. Do not mess around "under the hood" with a terminal and sudo. Make Synology earn your money. But if you want complete control, buy or build an x86_64/amd64 server, install Debian, and have at it. thanks for advices folks. It was indeed user error with being in a rush and blurred eyesight mistook "%" for "#" We are making progress. Appears that to retain permissions need root at both ends of rsync. Have keys working with ssh for users to NAS ( not helped by default permissions for .ssh files being wrong) and can su to root so now need to get ssh working with keys with no passphrase for root and all should be good. further to this if it helps anybody can start sshd with -d switch and at same time do client with -vv switch then can see where is falling down. Having telnet available helps if break sshd_config can still telnet and mend it. mick rsync is a quick & dirty backup tactic but it's got limitations. 1) files may stay around forever in the backup even if you've deleted them from your main computer because you don't need them. 2) you only have one copy of a file and that only lasts until the next rsync. This limits your ability to restore from a backup before it is overwritten. Using a real backup program, which can run on you main computer to backup to the NAS, lets you define a retention policy so files no longer needed can be purged while you have multiple backups of files you are currently working on. rsync is not a good substitute for backups.
Re: rsync to NAS for backup
On Thu, Feb 18, 2021 at 07:21:01PM +0200, Andrei POPESCU wrote: [...] > In my opinion the point still stands, rsync (by itself) has significant > limitations as a backup program, which is probably also the reason why > several backup programs using rsync exist. Care to name some of those limitations? (of course, rsync /is not/ a backup program in itself, but lets you build one with 10-30 lines around it). Cheers -- t signature.asc Description: Digital signature
Re: rsync to NAS for backup
On Thu, Feb 18, 2021 at 06:59:03PM +0200, Teemu Likonen wrote: > * 2021-02-18 11:13:25-0500, Gary Dale wrote: > > > rsync is a quick & dirty backup tactic but it's got limitations. > > > > 1) files may stay around forever in the backup even if you've deleted > > them from your main computer because you don't need them. > > > > 2) you only have one copy of a file and that only lasts until the next > > rsync. This limits your ability to restore from a backup before it is > > overwritten. > > > rsync is not a good substitute for backups. No, it's not. It is a fantastic tool for backups :-) > Rsync is great backup program with "--link-dest" option. Here is the > idea in simplified code: [...] Absolutely. Time travel! Actually, I've implemented this at a customer's place. They were delighted. Where rsync shows some weaknesses is on big, fat files (think videos, one or several GB). Really huge directories (tens to hundreds of TB) were once a challenge, too, but I hear that they refined the scanning part in the meantime. No direct experience, though. And, oh, Gary: if you want to delete files which disappeared in the source, check out the --delete option. But this time-staggered backup with --link-dest is really great. Cheers -- t signature.asc Description: Digital signature
Re: rsync to NAS for backup
On Jo, 18 feb 21, 18:59:03, Teemu Likonen wrote: > * 2021-02-18 11:13:25-0500, Gary Dale wrote: > > > rsync is a quick & dirty backup tactic but it's got limitations. > > > > 1) files may stay around forever in the backup even if you've deleted > > them from your main computer because you don't need them. > > > > 2) you only have one copy of a file and that only lasts until the next > > rsync. This limits your ability to restore from a backup before it is > > overwritten. > > > rsync is not a good substitute for backups. > > Rsync is great backup program with "--link-dest" option. Here is the > idea in simplified code: [...] > With that sort of code every backup is a new complete directory tree in > a time stamped directory. If files have not changed since the latest > backup "--link-dest" creates hard links. Old backups can be deleted by > removing old (or any) directory tree. The trees don't depend on each > other. > > What else do we need? OK, some people may need compression but usually > hard disk space is cheap. In my opinion the point still stands, rsync (by itself) has significant limitations as a backup program, which is probably also the reason why several backup programs using rsync exist. Kind regards, Andrei -- http://wiki.debian.org/FAQsFromDebianUser signature.asc Description: PGP signature
Re: rsync to NAS for backup
* 2021-02-18 11:13:25-0500, Gary Dale wrote:
> rsync is a quick & dirty backup tactic but it's got limitations.
>
> 1) files may stay around forever in the backup even if you've deleted
> them from your main computer because you don't need them.
>
> 2) you only have one copy of a file and that only lasts until the next
> rsync. This limits your ability to restore from a backup before it is
> overwritten.
> rsync is not a good substitute for backups.
Rsync is great backup program with "--link-dest" option. Here is the
idea in simplified code:
cd "$destination"
new=$(date +%Y%m%dT%H%M%S%z)
if mkdir "$new" && \
rsync -aHAX --link-dest=latest_backup [...] \
"$source" "$new"
then
# Recreate symlink to point to the new backup directory.
ln -sfn "$new" latest_backup
# Delete old backups.
find . -mindepth 1 -maxdepth 1 -type d -mtime +365 \
-exec /bin/rm -fr -- {} +
fi
With that sort of code every backup is a new complete directory tree in
a time stamped directory. If files have not changed since the latest
backup "--link-dest" creates hard links. Old backups can be deleted by
removing old (or any) directory tree. The trees don't depend on each
other.
What else do we need? OK, some people may need compression but usually
hard disk space is cheap.
--
/// Teemu Likonen - .-.. https://www.iki.fi/tlikonen/
// OpenPGP: 4E1055DC84E9DFF613D78557719D69D324539450
signature.asc
Description: PGP signature
Re: rsync to NAS for backup
On 2021-02-18 16:13, Gary Dale wrote: On 2021-02-18 10:57, mick crane wrote: On 2021-02-15 12:39, mick crane wrote: On 2021-02-13 19:20, David Christensen wrote: On 2021-02-13 01:27, mick crane wrote: I made a mistake and instead of getting a PC for backup I got a NAS. I'm struggling to get to grips with it. If rsync from PC to NAS NAS changes the owner/group of files to me/users which is probably no good for backing up. There's that problem then another that it won't let me login as root. I asked on Synology forum but not getting a lot of joy. https://community.synology.com/enu/forum/1/post/141137 Anybody used these things can advise ? What is the model of the Synology NAS? What options -- CPU, memory, disks, bays, interfaces, PSU, whatever? Support page URL? Reading the forum post, it sounds like you damaged the sudoers file. The fix would appear to be doing a Mode 2 reset per Synology's instructions: https://www.synology.com/en-global/knowledgebase/DSM/tutorial/General_Setup/How_to_reset_my_Synology_NAS Once the NAS has been reset, figure out how to meet your needs within the framework provided by Synology. Follow the User Guide. Follow the Admin Guide. Do not mess around "under the hood" with a terminal and sudo. Make Synology earn your money. But if you want complete control, buy or build an x86_64/amd64 server, install Debian, and have at it. thanks for advices folks. It was indeed user error with being in a rush and blurred eyesight mistook "%" for "#" We are making progress. Appears that to retain permissions need root at both ends of rsync. Have keys working with ssh for users to NAS ( not helped by default permissions for .ssh files being wrong) and can su to root so now need to get ssh working with keys with no passphrase for root and all should be good. further to this if it helps anybody can start sshd with -d switch and at same time do client with -vv switch then can see where is falling down. Having telnet available helps if break sshd_config can still telnet and mend it. mick rsync is a quick & dirty backup tactic but it's got limitations. 1) files may stay around forever in the backup even if you've deleted them from your main computer because you don't need them. 2) you only have one copy of a file and that only lasts until the next rsync. This limits your ability to restore from a backup before it is overwritten. Using a real backup program, which can run on you main computer to backup to the NAS, lets you define a retention policy so files no longer needed can be purged while you have multiple backups of files you are currently working on. rsync is not a good substitute for backups. OK, I didn't do this before. Am I right in thinking that BackupNinja keeps a local directory with list of files that need to be backed up and then rsyncs those files to remote directory ? mick -- Key ID4BFEBB31
Re: rsync to NAS for backup
On 2021-02-15 12:39, mick crane wrote: On 2021-02-13 19:20, David Christensen wrote: On 2021-02-13 01:27, mick crane wrote: I made a mistake and instead of getting a PC for backup I got a NAS. I'm struggling to get to grips with it. If rsync from PC to NAS NAS changes the owner/group of files to me/users which is probably no good for backing up. There's that problem then another that it won't let me login as root. I asked on Synology forum but not getting a lot of joy. https://community.synology.com/enu/forum/1/post/141137 Anybody used these things can advise ? What is the model of the Synology NAS? What options -- CPU, memory, disks, bays, interfaces, PSU, whatever? Support page URL? Reading the forum post, it sounds like you damaged the sudoers file. The fix would appear to be doing a Mode 2 reset per Synology's instructions: https://www.synology.com/en-global/knowledgebase/DSM/tutorial/General_Setup/How_to_reset_my_Synology_NAS Once the NAS has been reset, figure out how to meet your needs within the framework provided by Synology. Follow the User Guide. Follow the Admin Guide. Do not mess around "under the hood" with a terminal and sudo. Make Synology earn your money. But if you want complete control, buy or build an x86_64/amd64 server, install Debian, and have at it. thanks for advices folks. It was indeed user error with being in a rush and blurred eyesight mistook "%" for "#" We are making progress. Appears that to retain permissions need root at both ends of rsync. Have keys working with ssh for users to NAS ( not helped by default permissions for .ssh files being wrong) and can su to root so now need to get ssh working with keys with no passphrase for root and all should be good. further to this if it helps anybody can start sshd with -d switch and at same time do client with -vv switch then can see where is falling down. Having telnet available helps if break sshd_config can still telnet and mend it. mick -- Key ID4BFEBB31
Re: rsync to NAS for backup
On 2021-02-13 19:20, David Christensen wrote: On 2021-02-13 01:27, mick crane wrote: I made a mistake and instead of getting a PC for backup I got a NAS. I'm struggling to get to grips with it. If rsync from PC to NAS NAS changes the owner/group of files to me/users which is probably no good for backing up. There's that problem then another that it won't let me login as root. I asked on Synology forum but not getting a lot of joy. https://community.synology.com/enu/forum/1/post/141137 Anybody used these things can advise ? What is the model of the Synology NAS? What options -- CPU, memory, disks, bays, interfaces, PSU, whatever? Support page URL? Reading the forum post, it sounds like you damaged the sudoers file. The fix would appear to be doing a Mode 2 reset per Synology's instructions: https://www.synology.com/en-global/knowledgebase/DSM/tutorial/General_Setup/How_to_reset_my_Synology_NAS Once the NAS has been reset, figure out how to meet your needs within the framework provided by Synology. Follow the User Guide. Follow the Admin Guide. Do not mess around "under the hood" with a terminal and sudo. Make Synology earn your money. But if you want complete control, buy or build an x86_64/amd64 server, install Debian, and have at it. thanks for advices folks. It was indeed user error with being in a rush and blurred eyesight mistook "%" for "#" We are making progress. Appears that to retain permissions need root at both ends of rsync. Have keys working with ssh for users to NAS ( not helped by default permissions for .ssh files being wrong) and can su to root so now need to get ssh working with keys with no passphrase for root and all should be good. mick -- Key ID4BFEBB31
Re: rsync to NAS for backup
On 2021-02-13 01:27, mick crane wrote: I made a mistake and instead of getting a PC for backup I got a NAS. I'm struggling to get to grips with it. If rsync from PC to NAS NAS changes the owner/group of files to me/users which is probably no good for backing up. There's that problem then another that it won't let me login as root. I asked on Synology forum but not getting a lot of joy. https://community.synology.com/enu/forum/1/post/141137 Anybody used these things can advise ? What is the model of the Synology NAS? What options -- CPU, memory, disks, bays, interfaces, PSU, whatever? Support page URL? Reading the forum post, it sounds like you damaged the sudoers file. The fix would appear to be doing a Mode 2 reset per Synology's instructions: https://www.synology.com/en-global/knowledgebase/DSM/tutorial/General_Setup/How_to_reset_my_Synology_NAS Once the NAS has been reset, figure out how to meet your needs within the framework provided by Synology. Follow the User Guide. Follow the Admin Guide. Do not mess around "under the hood" with a terminal and sudo. Make Synology earn your money. But if you want complete control, buy or build an x86_64/amd64 server, install Debian, and have at it. David
Re: rsync to NAS for backup
On Sat, 13 Feb 2021 09:27:54 + mick crane wrote: > I made a mistake and instead of getting a PC for backup I got a NAS. > I'm struggling to get to grips with it. > If rsync from PC to NAS NAS changes the owner/group of files to > me/users which is probably no good for backing up. Can you set up a network file system such as NFS on the NAS, then tar your data to the NAS? Can you install a backup server such as amanda on the NAS? -- Does anybody read signatures any more? https://charlescurley.com https://charlescurley.com/blog/
Re: rsync to NAS for backup
Is there an alternative if you want an incremental backup? Obviously you could use tar-ed archives with unprivileged permissions. If you did, you would get a huge network overhead. thks Toni Mas GPG 3F42A21D84D7E950 Sent with ProtonMail Secure Email. ‐‐‐ Original Message ‐‐‐ En dissabte 13 de febrer de 2021 a les 13:50, didier gaumet va escriure: > Hello, > > Disclaimer: I do not use and am not familiar with Sinology hardware and > software and generally speaking, I am not knowledgeable in networking > > I would say that: > > - the owner:group names of a file on the PC you backup and the > owner:group names of the backup files on the synology files might be > different, even if you try to maintain ownership and rights. What really > counts here are owner:group identifiers (UID:GID). Bob_user:Bob_group on > your PC might equate to Alice_user:John_group on your NAS. Upon > restoration that would be reversed to Bob_user:Bob_group. > That would be typical without something like a LDAP server. > > - SSH root login seems to be discouraged for security reasons. Sinology > probably adhere to this principle and the appropriate way to do what you > want would probably be to access a shell on the Synology software to > issue a sudo or su -c command. > > - editing /etc/sudoers is generally done via the visudo command > - if that is of interest to you, there is a way to install Debian in > chroot on your NAS > signature.asc Description: OpenPGP digital signature
Re: rsync to NAS for backup
Hello, Disclaimer: I do not use and am not familiar with Sinology hardware and software and generally speaking, I am not knowledgeable in networking I would say that: - the owner:group names of a file on the PC you backup and the owner:group names of the backup files on the synology files might be different, even if you try to maintain ownership and rights. What really counts here are owner:group identifiers (UID:GID). Bob_user:Bob_group on your PC might equate to Alice_user:John_group on your NAS. Upon restoration that would be reversed to Bob_user:Bob_group. That would be typical without something like a LDAP server. - SSH root login seems to be discouraged for security reasons. Sinology probably adhere to this principle and the appropriate way to do what you want would probably be to access a shell on the Synology software to issue a sudo or su -c command. - editing /etc/sudoers is generally done via the visudo command - if that is of interest to you, there is a way to install Debian in chroot on your NAS
rsync to NAS for backup
I made a mistake and instead of getting a PC for backup I got a NAS. I'm struggling to get to grips with it. If rsync from PC to NAS NAS changes the owner/group of files to me/users which is probably no good for backing up. There's that problem then another that it won't let me login as root. I asked on Synology forum but not getting a lot of joy. https://community.synology.com/enu/forum/1/post/141137 Anybody used these things can advise ? mick -- Key ID4BFEBB31
