Re: [gentoo-user] rsync backup system
On Thu, Feb 25, 2010 at 17:11, Ward Poelmans wpoel...@gmail.com wrote: On Thu, Feb 25, 2010 at 16:41, Alan McKinnon alan.mckin...@gmail.com wrote: A much better way is to run a dedicated agent on the client. If the server needs to schedule backups, it can ask the agent to do so using regular tcp traffic. The client can then do it's backup and rsync it over to the server when it's done, and that push can be done as a regular user on both ends. The actual backing up on the client must be done by root of course, no other user has the necessary access. If anyone is still interested, i had some time and this is what i did: On the client: rsync -a -X -b --backup-dir=../backup.0/ --link-dest=../backup.0/ /home/ward backupserver:Backup-Laptop/backup.cur/ ssh backupserver /home/ward/shiftbackups.sh This makes a directory backup.cur on the backupserver with a full backup in it, but it's exactly only a incremental backup as it hardlinks from backup.0 (the previous backup). The script shiftbackups.sh moves backup.0 to backup.1 and backup.cur to backup.0 and so on... This does more or less exactly what i wanted. Regards, Ward
Re: [gentoo-user] rsync backup system
On Thu, Feb 25, 2010 at 18:50, daid kahl daid...@gmail.com wrote: As a simple idea, cron task starts rsnapshot configured however. When this is done, backup is tarballed, and tarball is given as like, say, 440 permissions, where users are in some useful 'backup' group, then while tarball can be read to be passed across server, if tarball is extracted, user has no more privs then they have on the system anyway (I'm not saying chmod -R). Then local tarball can be removed or whatever. It's not a bad idea, but you need enough free space on the client to backup the entire system (which for me is not the case). Secondly, every backup you do is a full backup as rsnapshot needs to access a backup todo a incremental backup. You could mess around with something like sshfs but's it's not great either. A straight rsync between client and server could do it but it would suprise me if this doesn't already exist in some form. Regards, Ward
Re: [gentoo-user] rsync backup system
On 26 February 2010 22:23, Ward Poelmans wpoel...@gmail.com wrote: On Thu, Feb 25, 2010 at 18:50, daid kahl daid...@gmail.com wrote: As a simple idea, cron task starts rsnapshot configured however. When this is done, backup is tarballed, and tarball is given as like, say, 440 permissions, where users are in some useful 'backup' group, then while tarball can be read to be passed across server, if tarball is extracted, user has no more privs then they have on the system anyway (I'm not saying chmod -R). Then local tarball can be removed or whatever. It's not a bad idea, but you need enough free space on the client to backup the entire system (which for me is not the case). Secondly, every backup you do is a full backup as rsnapshot needs to access a backup todo a incremental backup. You could mess around with something like sshfs but's it's not great either. A straight rsync between client and server could do it but it would suprise me if this doesn't already exist in some form. Regards, Ward Thanks for the feedback. For now, as you may easily guess, this case does not apply to me personally since I mostly just admin my own personal machine. But I think you raise very relevant difficulties with my suggestion for a practical administrative case for multiple machines. ~daid
Re: [gentoo-user] rsync backup system
On Wed, Feb 24, 2010 at 21:51, Neil Bothwick n...@digimed.co.uk wrote: It looks interesting, and no program is that good that alternatives should never be considered, but I really like the way BaclupPC works. Everything is handled by the server, all you need to do on each client is copy the backuppc user's public key to /root/.ssh/authorized_keys. Well, that's one of the things i don't like. Is there backup software where the client does the backup to the server and not the server fetching the backup from the client? I can't find a good way to take regular backup's from laptop that come and go. Regards, Ward
Re: [gentoo-user] rsync backup system
On Thursday 25 February 2010 17:15:36 Ward Poelmans wrote: On Wed, Feb 24, 2010 at 21:51, Neil Bothwick n...@digimed.co.uk wrote: It looks interesting, and no program is that good that alternatives should never be considered, but I really like the way BaclupPC works. Everything is handled by the server, all you need to do on each client is copy the backuppc user's public key to /root/.ssh/authorized_keys. Well, that's one of the things i don't like. Is there backup software where the client does the backup to the server and not the server fetching the backup from the client? I can't find a good way to take regular backup's from laptop that come and go. Never mind the massive security issues resulting from the backup server logging into the clients with a passphrase-less key, AS ROOT. Which means you now have to open up root logins over ssh on the clients. And someone gets into your backup server, BANG! instant pwnage of every single machine on your network. Heck, you don't even have to try and compromise the local root account, you already have full unfettered access to everything anyway. Worse, I'll bet the server software runs as an unpriviledged user, so you can just bypass the bit where you have to compromise root there as well. It all looks like a classic case of sacrificing any and all security in the name of mere convenience. I have the same running battle at work, some idiot (probably in marketing getting a kickback) wants me to run a frigging JAVA backup app on my mission critical servers, the ones with 500+ users on them most of them chancers of the first order. They didn't understand the irony when I suggested I should just dispense with passwords and keys altogether and chmod -R 777 / A much better way is to run a dedicated agent on the client. If the server needs to schedule backups, it can ask the agent to do so using regular tcp traffic. The client can then do it's backup and rsync it over to the server when it's done, and that push can be done as a regular user on both ends. The actual backing up on the client must be done by root of course, no other user has the necessary access. -- alan dot mckinnon at gmail dot com
Re: [gentoo-user] rsync backup system
On Thu, 25 Feb 2010 16:15:36 +0100, Ward Poelmans wrote: It looks interesting, and no program is that good that alternatives should never be considered, but I really like the way BaclupPC works. Everything is handled by the server, all you need to do on each client is copy the backuppc user's public key to /root/.ssh/authorized_keys. Well, that's one of the things i don't like. Is there backup software where the client does the backup to the server and not the server fetching the backup from the client? I can't find a good way to take regular backup's from laptop that come and go. You can manually start a backup using the BackupPC web interface, but the automatic backup thing works with laptops, the server just waits until the laptop appears on the network. -- Neil Bothwick Synonym: a word you use when you can't spell the other one. signature.asc Description: PGP signature
Re: [gentoo-user] rsync backup system
On Thu, Feb 25, 2010 at 16:41, Alan McKinnon alan.mckin...@gmail.com wrote: A much better way is to run a dedicated agent on the client. If the server needs to schedule backups, it can ask the agent to do so using regular tcp traffic. The client can then do it's backup and rsync it over to the server when it's done, and that push can be done as a regular user on both ends. The actual backing up on the client must be done by root of course, no other user has the necessary access. Sounds great. Is there any software that works this way? Ward
Re: [gentoo-user] rsync backup system
On 26 February 2010 01:11, Ward Poelmans wpoel...@gmail.com wrote: On Thu, Feb 25, 2010 at 16:41, Alan McKinnon alan.mckin...@gmail.com wrote: A much better way is to run a dedicated agent on the client. If the server needs to schedule backups, it can ask the agent to do so using regular tcp traffic. The client can then do it's backup and rsync it over to the server when it's done, and that push can be done as a regular user on both ends. The actual backing up on the client must be done by root of course, no other user has the necessary access. Sounds great. Is there any software that works this way? Ward Sounds more or less like cron tasks and rsnapshot to me (can use other rsync scripts of course, but this one is nice to me anyway, and someone else mentioned it earlier in the thread). I'm not sure off hand I have a good way for it to be initialized from the server end, but if it's a backup, it might as well run on a local cron anyway rather than needing an external call. As a simple idea, cron task starts rsnapshot configured however. When this is done, backup is tarballed, and tarball is given as like, say, 440 permissions, where users are in some useful 'backup' group, then while tarball can be read to be passed across server, if tarball is extracted, user has no more privs then they have on the system anyway (I'm not saying chmod -R). Then local tarball can be removed or whatever. And call me silly for not reading documentation or assuming, but I was very happy last night when I realized system rescue CD includes rsnapshot already! ~daid
Re: [gentoo-user] rsync backup system
On Thu, 25 Feb 2010 17:41:13 +0200, Alan McKinnon wrote: And someone gets into your backup server, BANG! instant pwnage of every single machine on your network. Heck, you don't even have to try and compromise the local root account, you already have full unfettered access to everything anyway. Which is why you don't allow access to the backup server from outside of the network, and restrict root access from inside. Because backups are initiated from the server, it doesn't actually need any ports open to do its job, although a web server is needed to run the user interface (which isn't necessary). The ebuild sets up a separate instance of Apache just for this, so even if you are already running Apache on the backup server (which is a crazy idea to start with) compromising that won't get you into the backups. Worse, I'll bet the server software runs as an unpriviledged user, so you can just bypass the bit where you have to compromise root there as well. You lose :P The server runs as a restricted user, with no login shell. -- Neil Bothwick WinErr 042: Virus error - A virus has been activated in a dos-box. The virus, however, requires Windows. All tasks will automatically be closed and the virus will be activated again. signature.asc Description: PGP signature
Re: [gentoo-user] rsync backup system
A much better way is to run a dedicated agent on the client. If the server needs to schedule backups, it can ask the agent to do so using regular tcp traffic. The client can then do it's backup and rsync it over to the server when it's done, and that push can be done as a regular user on both ends. The actual backing up on the client must be done by root of course, no other user has the necessary access. Sounds great. Is there any software that works this way? Ward I'd like to know too. :) - Grant
Re: [gentoo-user] rsync backup system
How is BackupPC to set up? Is it a whole new world to explore, or can it be set up quickly and easily? It takes a little while to get the hang of how the config files work, but once you get it it takes no work at all. Restoring is as simple as selecting the files you want in a browser and pressing a button. -- Neil Bothwick Has anyone tried backupninja? There is a new ebuild for it. https://labs.riseup.net/code/projects/show/backupninja/ Is BackupPC too excellent to consider an alternative? I'm going to set up one of these backup systems in the next few weeks. - Grant
Re: [gentoo-user] rsync backup system
On Wed, 24 Feb 2010 11:02:42 -0800, Grant wrote: Has anyone tried backupninja? There is a new ebuild for it. https://labs.riseup.net/code/projects/show/backupninja/ Is BackupPC too excellent to consider an alternative? I'm going to set up one of these backup systems in the next few weeks. It looks interesting, and no program is that good that alternatives should never be considered, but I really like the way BaclupPC works. Everything is handled by the server, all you need to do on each client is copy the backuppc user's public key to /root/.ssh/authorized_keys. -- Neil Bothwick The dark ages were caused by the Y1K problem. signature.asc Description: PGP signature
Re: [gentoo-user] rsync backup system
4. Should I be comfortable running the entire sync operation every night, or am I jeopardizing the longevity of my HDs? This is a joke. I should apologize and explain this better. If you bought a fancy expensive hard drive then it's probably designed for extremely heavy use and comes with nice coverage by the guy that sold it to you. But since you ask this question I'll assume that it's not the case. Ok so you bought a relatively cheap heard drive. But the question seems irrelevant. Because these are cheap, mass-produced hard drives you could do absolutely nothing with them and they could still die tomorrow. That's the price you pay for cheaper drives. Cheap hard drives are like life. Life is cheap. You could take all kinds of precautions and still die tomorrow. But your drive is cheap and easily replaced (esp. if it's still under warranty). Your data on the other hand is not. I'd rather have a dead drive with the data backed up than a dead drive with no backup. Drives are so cheap nowadays it's a non-issue. I'm actually hoping my 2-year old drive dies soon because it will give me an excuse to go out and buy a *bigger* one for the same price. But I feel comfortable with that because I keep backups. Thank you for the clarification. Which are the fancy expensive hard drives? - Grant
Re: [gentoo-user] rsync backup system
2. Some of the files I back up only allow root to read. I can run rsync as root on each system, but I don't allow root logins. This means in order to rsync the second sync system with the first sync system, I must run the rsync command from the first sync system. This means I have to run rsyncd on the second sync system in addition to the first sync system. I'd rather only run one instance of rsyncd. Can anyone think of another option? Allow root logins only with a key, set up a specific user on the backup server to run the backup tasks and add that users key to the authorized_users file on the machine to be backed up. 3. The rsync process always completes with rsync error: some files/attrs were not transferred. How can I get more information about which files this pertains to? Run rsync with the verbose option and direct stdout and stderr to files. 5. If I end up with filesystem corruption on the SRC system, will that corruption transfer over to the DST system during an rsync, or will the transfer just fail? If data is corrupted, that will be backed up. If the filesystem corruption causes read errors, rsync will bail out. 6. Can I run rsyncd on a system facing the internet without fear? Yes, as long as the rsync ports are closed in your router. Instead of all this, I'd recommend BackupPC. It handles all of your issues and more and is efficient at backing up multiple machines. You could run one BackupPC server and then rsync its store to the backup backup server. The latest version in portage is old, get the 3.x ebuild from bgo. How is BackupPC to set up? Is it a whole new world to explore, or can it be set up quickly and easily? - Grant
Re: [gentoo-user] rsync backup system
On Thu, Nov 19, 2009 at 19:19, Grant emailgr...@gmail.com wrote: BackupPC does look pretty good. Would anyone recommend I *don't* can this whole thing and set up BackupPC instead? I recommend you take a look at rsnapshot instead of pure rsync. Ward
Re: [gentoo-user] rsync backup system
On Fri, 2009-11-20 at 08:03 -0800, Grant wrote: Thank you for the clarification. Which are the fancy expensive hard drives? The SAS drives that run at 15k RPM and cost $2-3 USD per GB. As opposed to your run of the mill 7200RPM SATA drive that costs pennies per GB.
Re: [gentoo-user] rsync backup system
On Fri, 20 Nov 2009 08:05:27 -0800, Grant wrote: How is BackupPC to set up? Is it a whole new world to explore, or can it be set up quickly and easily? It takes a little while to get the hang of how the config files work, but once you get it it takes no work at all. Restoring is as simple as selecting the files you want in a browser and pressing a button. -- Neil Bothwick LISP: Lots of Infuriating Silly Parentheses signature.asc Description: PGP signature
[gentoo-user] rsync backup system
I just finished an rsync backup system that works like this: Each of 4 Gentoo systems contains a folder called backup which contains symlinks to local files and folders for backup. 2 of the systems contain a folder called sync which contains the contents of the backup folder for each of the 4 systems. 3 of the systems rsync with one of the sync systems, and that system rsyncs with the other sync system. I've got a few questions for you guys about this. 1. I back up the entire /etc folder of each system, and some files have read-only permissions. This means I get permission denied when I try to rsync them. How would you handle this? 2. Some of the files I back up only allow root to read. I can run rsync as root on each system, but I don't allow root logins. This means in order to rsync the second sync system with the first sync system, I must run the rsync command from the first sync system. This means I have to run rsyncd on the second sync system in addition to the first sync system. I'd rather only run one instance of rsyncd. Can anyone think of another option? 3. The rsync process always completes with rsync error: some files/attrs were not transferred. How can I get more information about which files this pertains to? 4. Should I be comfortable running the entire sync operation every night, or am I jeopardizing the longevity of my HDs? 5. If I end up with filesystem corruption on the SRC system, will that corruption transfer over to the DST system during an rsync, or will the transfer just fail? 6. Can I run rsyncd on a system facing the internet without fear? - Grant
Re: [gentoo-user] rsync backup system
On Thu, 19 Nov 2009 08:44:44 -0800, Grant wrote: 2. Some of the files I back up only allow root to read. I can run rsync as root on each system, but I don't allow root logins. This means in order to rsync the second sync system with the first sync system, I must run the rsync command from the first sync system. This means I have to run rsyncd on the second sync system in addition to the first sync system. I'd rather only run one instance of rsyncd. Can anyone think of another option? Allow root logins only with a key, set up a specific user on the backup server to run the backup tasks and add that users key to the authorized_users file on the machine to be backed up. 3. The rsync process always completes with rsync error: some files/attrs were not transferred. How can I get more information about which files this pertains to? Run rsync with the verbose option and direct stdout and stderr to files. 5. If I end up with filesystem corruption on the SRC system, will that corruption transfer over to the DST system during an rsync, or will the transfer just fail? If data is corrupted, that will be backed up. If the filesystem corruption causes read errors, rsync will bail out. 6. Can I run rsyncd on a system facing the internet without fear? Yes, as long as the rsync ports are closed in your router. Instead of all this, I'd recommend BackupPC. It handles all of your issues and more and is efficient at backing up multiple machines. You could run one BackupPC server and then rsync its store to the backup backup server. The latest version in portage is old, get the 3.x ebuild from bgo. -- Neil Bothwick Of all the people I've met you're certainly one of them signature.asc Description: PGP signature
Re: [gentoo-user] rsync backup system
On Thu, 2009-11-19 at 08:44 -0800, Grant wrote: I just finished an rsync backup system that works like this: Each of 4 Gentoo systems contains a folder called backup which contains symlinks to local files and folders for backup. 2 of the systems contain a folder called sync which contains the contents of the backup folder for each of the 4 systems. 3 of the systems rsync with one of the sync systems, and that system rsyncs with the other sync system. I've got a few questions for you guys about this. 1. I back up the entire /etc folder of each system, and some files have read-only permissions. This means I get permission denied when I try to rsync them. How would you handle this? I've never had this problem. You should only need read permissions to copy a file: $ touch this $ chmod 0400 this $ rsync -a this that $ /bin/cp -f /etc/issue this $ chmod 0400 this $ rsync -a this that $ ls -l this that 4.0K -r 1 percy users 3 Nov 19 12:38 that 4.0K -r 1 percy users 3 Nov 19 12:38 this 2. Some of the files I back up only allow root to read. I can run rsync as root on each system, but I don't allow root logins. This means in order to rsync the second sync system with the first sync system, I must run the rsync command from the first sync system. This means I have to run rsyncd on the second sync system in addition to the first sync system. I'd rather only run one instance of rsyncd. Can anyone think of another option? Well if #1 was not run as root then this is where you have problems. And you will continue to have problems if you don't copy as root or some user who has at least read access to all the files on each source and can also preserve ownership/permissions on the target system. Like... root. ssh. Why are you using rsyncd anyway? AFAIK rsyncd is not encrypted. You can allow a non-root user to ssh in and run rsync as root via a proxy command. You can also configure sshd to only allow root to log in and execute a particular command (such as rsync). See PermitRootLogin and ForcedCommand under ssh_config(5) 3. The rsync process always completes with rsync error: some files/attrs were not transferred. How can I get more information about which files this pertains to? It should spit it out to stderr. If you using -v they probably scrolled by. Redirect stderr to a file. If you are running rsync as non-root then it can't preserve ownership and some perms. If you are using extended attributes/ACLs and did not tell rsync to use them (or comple support in) or if the target system does not support them then they will not be transferred. 4. Should I be comfortable running the entire sync operation every night, or am I jeopardizing the longevity of my HDs? This is a joke. 5. If I end up with filesystem corruption on the SRC system, will that corruption transfer over to the DST system during an rsync, or will the transfer just fail? If the filesystem returns an error to rsync then rsync will error out. Simple as that. If the file is currupt (not the filesystem) then, standard garbage-in/garbage-out rules apply. 6. Can I run rsyncd on a system facing the internet without fear? No. Rsyncd is not encrypted (see above) also the authentication is weak (see the man page). Use ssh. It's more secure and had better lock-down mechanisms. Better yet, use a VPN and ssh through the VPN (double authentication (and encryption)).
Re: [gentoo-user] rsync backup system
On Thu, 2009-11-19 at 12:58 -0500, Albert Hopkins wrote: 4. Should I be comfortable running the entire sync operation every night, or am I jeopardizing the longevity of my HDs? This is a joke. I should apologize and explain this better. If you bought a fancy expensive hard drive then it's probably designed for extremely heavy use and comes with nice coverage by the guy that sold it to you. But since you ask this question I'll assume that it's not the case. Ok so you bought a relatively cheap heard drive. But the question seems irrelevant. Because these are cheap, mass-produced hard drives you could do absolutely nothing with them and they could still die tomorrow. That's the price you pay for cheaper drives. Cheap hard drives are like life. Life is cheap. You could take all kinds of precautions and still die tomorrow. But your drive is cheap and easily replaced (esp. if it's still under warranty). Your data on the other hand is not. I'd rather have a dead drive with the data backed up than a dead drive with no backup. Drives are so cheap nowadays it's a non-issue. I'm actually hoping my 2-year old drive dies soon because it will give me an excuse to go out and buy a *bigger* one for the same price. But I feel comfortable with that because I keep backups.
Re: [gentoo-user] rsync backup system
2. Some of the files I back up only allow root to read. I can run rsync as root on each system, but I don't allow root logins. This means in order to rsync the second sync system with the first sync system, I must run the rsync command from the first sync system. This means I have to run rsyncd on the second sync system in addition to the first sync system. I'd rather only run one instance of rsyncd. Can anyone think of another option? Allow root logins only with a key, set up a specific user on the backup server to run the backup tasks and add that users key to the authorized_users file on the machine to be backed up. 3. The rsync process always completes with rsync error: some files/attrs were not transferred. How can I get more information about which files this pertains to? Run rsync with the verbose option and direct stdout and stderr to files. 5. If I end up with filesystem corruption on the SRC system, will that corruption transfer over to the DST system during an rsync, or will the transfer just fail? If data is corrupted, that will be backed up. If the filesystem corruption causes read errors, rsync will bail out. 6. Can I run rsyncd on a system facing the internet without fear? Yes, as long as the rsync ports are closed in your router. Instead of all this, I'd recommend BackupPC. It handles all of your issues and more and is efficient at backing up multiple machines. You could run one BackupPC server and then rsync its store to the backup backup server. The latest version in portage is old, get the 3.x ebuild from bgo. BackupPC does look pretty good. Would anyone recommend I *don't* can this whole thing and set up BackupPC instead? - Grant
