Re: [gentoo-user] {OT} Are push backups flawed?
On 11/14/11 20:54, Grant wrote: If you're intent on making a two-stage pull work; you can do it by creating a 'backups' user on your servers, and then using filesystem ACLs to grant backups+r to every file/directory you want to back up. That way, an attacker on the backup server can't decide to peruse the rest of your stuff. I like that. So use ACLs to grant access to the backups instead of using ownership/permissions so that the ownership/permissions stay intact. I've never used ACLs. Do they override ownership/permissions? In other words, if the ACL specifies backups+r to a file owned by root that is chmod 700, backups can read it anyway? Yup, they work like Windows ACLs if you've used those. You can grant one user read permission without affecting anything else. The '700' mode doesn't really make sense anymore after you apply an ACL.. the whole permissions-as-bits concept gets highly convoluted[1] but if you just want to add read access for one user it's easy. You can use setfacl to add permissions, and double-check with getfacl that they do what you think they do. The examples in `man setfacl` are pretty easy to understand. The easiest method, though, is to just add a third stage. Either move the backups on the backup server to another directory after the backup job completes, or sync/burn/whatever them off-site. In this case the backup server can't access anything you don't give it, and the individual servers can't trash their backed-up data. I don't see how that could work in an automated fashion. Could you give me an example? We do push backups to one server, backup1, every night. Then, every day, backup1 syncs to another server, backup2. The individuals servers have no access to backup2, and it's physically separate from backup1. I make physical, removable, backups of backup2 every once in a while, but not as often as I should. [1] http://www.suse.de/~agruen/acl/linux-acls/online/
Re: [gentoo-user] {OT} Are push backups flawed?
And if I pull, none of my backed-up systems are secure because anyone who breaks into the backup server has root read privileges on every backed-up system and will thereby gain full root privileges quickly. IMO that depends on whether you also backup the authentication-related files or not. Exclude them from backup, ensure different root passwords for all boxes, and now you can limit the infiltration. If you're pulling to the backup server, that backup server has to be able to log in to and read all files on the other servers. Including e.g. your swap partition and device files. What if I have each system save a copy of everything to be backed up from its own filesystem in a separate directory and change the ownership of everything in that directory so it can be read by an unprivileged backup user? Then I could have the backup server pull that copy from each system without giving it root access to each system. Can I somehow have the correct ownerships for the backup saved in a separate file for use during a restore? - Grant You could just as well use an NFS share with no_root_squash. It is really more a question of finding the right combination of tools to ensure proper separation of concern for server and client. In fact, I think we are intermixing three distinct problems: 1. (Possible) limitations of rdiff-backup with regard to untrusted backup servers or clients. The limitation is real unfortunately. All backups created by rdiff-backup more than a second ago can be deleted something like this: rdiff-backup --remove-older-than 1s backup@12.34.56.78::/path/to/backup 2. The purely technical question which file transfer protocols protect against write access from backup server to backup client and backup client to older backups on the server. rdiff-backup doesn't provide those sort of protections. Do any file transfer protocols? 3. The more or less organisational question what level of protection backups need and how fast security breaks have to be detected. I think backups should be very well protected and security breaks should not have to be immediately detected. - Grant I think push vs. pull is just a secondary concern with regard to the second question and has practically no relevance to the third one. Regards, Florian Philipp
Re: [gentoo-user] {OT} Are push backups flawed?
And if I pull, none of my backed-up systems are secure because anyone who breaks into the backup server has root read privileges on every backed-up system and will thereby gain full root privileges quickly. IMO that depends on whether you also backup the authentication-related files or not. Exclude them from backup, ensure different root passwords for all boxes, and now you can limit the infiltration. If you're pulling to the backup server, that backup server has to be able to log in to and read all files on the other servers. Including e.g. your swap partition and device files. What if I have each system save a copy of everything to be backed up from its own filesystem in a separate directory and change the ownership of everything in that directory so it can be read by an unprivileged backup user? You've just reinvented the push backup =) If separate-directory is on the same server, an attacker can log in and overwrite all of your files with zeros. Those zeros will be pulled to the backup server, destroying your backups. That's not the case at all. The zeros would be pulled to the backup server via rdiff-backup and saved as a new version in the repository. The backups would be safe. - Grant If separate-directory is on the backup server...
Re: [gentoo-user] {OT} Are push backups flawed?
It's out of scope for file transfer protocols; it's a daemon/system-local problem. Attach pre-event or post-event scripts serverside for any special munging or protections you'd like to apply. (Such as triggering an LVM snapshot, for example...) (sorry for the top post; in-line can be done in this client, but it's more cumbersome than I have time for atm...) ZZ On Nov 14, 2011 7:45 PM, Grant emailgr...@gmail.com wrote: And if I pull, none of my backed-up systems are secure because anyone who breaks into the backup server has root read privileges on every backed-up system and will thereby gain full root privileges quickly. IMO that depends on whether you also backup the authentication-related files or not. Exclude them from backup, ensure different root passwords for all boxes, and now you can limit the infiltration. If you're pulling to the backup server, that backup server has to be able to log in to and read all files on the other servers. Including e.g. your swap partition and device files. What if I have each system save a copy of everything to be backed up from its own filesystem in a separate directory and change the ownership of everything in that directory so it can be read by an unprivileged backup user? Then I could have the backup server pull that copy from each system without giving it root access to each system. Can I somehow have the correct ownerships for the backup saved in a separate file for use during a restore? - Grant You could just as well use an NFS share with no_root_squash. It is really more a question of finding the right combination of tools to ensure proper separation of concern for server and client. In fact, I think we are intermixing three distinct problems: 1. (Possible) limitations of rdiff-backup with regard to untrusted backup servers or clients. The limitation is real unfortunately. All backups created by rdiff-backup more than a second ago can be deleted something like this: rdiff-backup --remove-older-than 1s backup@12.34.56.78::/path/to/backup 2. The purely technical question which file transfer protocols protect against write access from backup server to backup client and backup client to older backups on the server. rdiff-backup doesn't provide those sort of protections. Do any file transfer protocols? 3. The more or less organisational question what level of protection backups need and how fast security breaks have to be detected. I think backups should be very well protected and security breaks should not have to be immediately detected. - Grant I think push vs. pull is just a secondary concern with regard to the second question and has practically no relevance to the third one. Regards, Florian Philipp
Re: [gentoo-user] {OT} Are push backups flawed?
It's out of scope for file transfer protocols; it's a daemon/system-local problem. Attach pre-event or post-event scripts serverside for any special munging or protections you'd like to apply. (Such as triggering an LVM snapshot, for example...) I must be going about this the wrong way. Am I the only one using automated backups? If not, how is it done properly? - Grant And if I pull, none of my backed-up systems are secure because anyone who breaks into the backup server has root read privileges on every backed-up system and will thereby gain full root privileges quickly. IMO that depends on whether you also backup the authentication-related files or not. Exclude them from backup, ensure different root passwords for all boxes, and now you can limit the infiltration. If you're pulling to the backup server, that backup server has to be able to log in to and read all files on the other servers. Including e.g. your swap partition and device files. What if I have each system save a copy of everything to be backed up from its own filesystem in a separate directory and change the ownership of everything in that directory so it can be read by an unprivileged backup user? Then I could have the backup server pull that copy from each system without giving it root access to each system. Can I somehow have the correct ownerships for the backup saved in a separate file for use during a restore? - Grant You could just as well use an NFS share with no_root_squash. It is really more a question of finding the right combination of tools to ensure proper separation of concern for server and client. In fact, I think we are intermixing three distinct problems: 1. (Possible) limitations of rdiff-backup with regard to untrusted backup servers or clients. The limitation is real unfortunately. All backups created by rdiff-backup more than a second ago can be deleted something like this: rdiff-backup --remove-older-than 1s backup@12.34.56.78::/path/to/backup 2. The purely technical question which file transfer protocols protect against write access from backup server to backup client and backup client to older backups on the server. rdiff-backup doesn't provide those sort of protections. Do any file transfer protocols? 3. The more or less organisational question what level of protection backups need and how fast security breaks have to be detected. I think backups should be very well protected and security breaks should not have to be immediately detected. - Grant I think push vs. pull is just a secondary concern with regard to the second question and has practically no relevance to the third one. Regards, Florian Philipp
Re: [gentoo-user] {OT} Are push backups flawed?
Then I could have the backup server pull that copy from each system without giving it root access to each system. Can I somehow have the correct ownerships for the backup saved in a separate file for use during a restore? If you're intent on making a two-stage pull work; you can do it by creating a 'backups' user on your servers, and then using filesystem ACLs to grant backups+r to every file/directory you want to back up. That way, an attacker on the backup server can't decide to peruse the rest of your stuff. I like that. So use ACLs to grant access to the backups instead of using ownership/permissions so that the ownership/permissions stay intact. I've never used ACLs. Do they override ownership/permissions? In other words, if the ACL specifies backups+r to a file owned by root that is chmod 700, backups can read it anyway? The easiest method, though, is to just add a third stage. Either move the backups on the backup server to another directory after the backup job completes, or sync/burn/whatever them off-site. In this case the backup server can't access anything you don't give it, and the individual servers can't trash their backed-up data. I don't see how that could work in an automated fashion. Could you give me an example? - Grant
Re: [gentoo-user] {OT} Are push backups flawed?
You identified a flaw in the system as you were using it. You're right, those are flaws. However, you can fix those flaws by applying some magic as a sysadmin. That's why several posts in the thread have mentioned versioning your backups in some fashion. I've mentioned lvm a couple times. I think someone else mentioned pulling the backup target's data to another locale, either via a pull from another server, or via something like a traditional incremental tape backup. You're getting the data off the original machines to a remote location, which is good. You identified a way the backed-up data could be tampered with, which is good. You just need to put in another (better) barrier to protect the data from being tampered with, or limit how much data is lost in such an event. ZZ On Nov 14, 2011 8:21 PM, Grant emailgr...@gmail.com wrote: It's out of scope for file transfer protocols; it's a daemon/system-local problem. Attach pre-event or post-event scripts serverside for any special munging or protections you'd like to apply. (Such as triggering an LVM snapshot, for example...) I must be going about this the wrong way. Am I the only one using automated backups? If not, how is it done properly? - Grant And if I pull, none of my backed-up systems are secure because anyone who breaks into the backup server has root read privileges on every backed-up system and will thereby gain full root privileges quickly. IMO that depends on whether you also backup the authentication-related files or not. Exclude them from backup, ensure different root passwords for all boxes, and now you can limit the infiltration. If you're pulling to the backup server, that backup server has to be able to log in to and read all files on the other servers. Including e.g. your swap partition and device files. What if I have each system save a copy of everything to be backed up from its own filesystem in a separate directory and change the ownership of everything in that directory so it can be read by an unprivileged backup user? Then I could have the backup server pull that copy from each system without giving it root access to each system. Can I somehow have the correct ownerships for the backup saved in a separate file for use during a restore? - Grant You could just as well use an NFS share with no_root_squash. It is really more a question of finding the right combination of tools to ensure proper separation of concern for server and client. In fact, I think we are intermixing three distinct problems: 1. (Possible) limitations of rdiff-backup with regard to untrusted backup servers or clients. The limitation is real unfortunately. All backups created by rdiff-backup more than a second ago can be deleted something like this: rdiff-backup --remove-older-than 1s backup@12.34.56.78::/path/to/backup 2. The purely technical question which file transfer protocols protect against write access from backup server to backup client and backup client to older backups on the server. rdiff-backup doesn't provide those sort of protections. Do any file transfer protocols? 3. The more or less organisational question what level of protection backups need and how fast security breaks have to be detected. I think backups should be very well protected and security breaks should not have to be immediately detected. - Grant I think push vs. pull is just a secondary concern with regard to the second question and has practically no relevance to the third one. Regards, Florian Philipp
Re: [gentoo-user] {OT} Are push backups flawed?
You identified a flaw in the system as you were using it. You're right, those are flaws. However, you can fix those flaws by applying some magic as a sysadmin. That's why several posts in the thread have mentioned versioning your backups in some fashion. I've mentioned lvm a couple times. I thought versioning meant that you could roll back to a previous version. rdiff-backup provides that. I think someone else mentioned pulling the backup target's data to another locale, either via a pull from another server, or via something like a traditional incremental tape backup. So the systems push to the backup server and a second backup server pulls from the first backup server? Should the second backup server use rdiff-backup against the rdiff-backup repository on the first backup server? I think I've read that's not a good idea. What does everybody else do? I feel like the first person to ever attempt secure automated backups. - Grant You're getting the data off the original machines to a remote location, which is good. You identified a way the backed-up data could be tampered with, which is good. You just need to put in another (better) barrier to protect the data from being tampered with, or limit how much data is lost in such an event. ZZ On Nov 14, 2011 8:21 PM, Grant emailgr...@gmail.com wrote: It's out of scope for file transfer protocols; it's a daemon/system-local problem. Attach pre-event or post-event scripts serverside for any special munging or protections you'd like to apply. (Such as triggering an LVM snapshot, for example...) I must be going about this the wrong way. Am I the only one using automated backups? If not, how is it done properly? - Grant And if I pull, none of my backed-up systems are secure because anyone who breaks into the backup server has root read privileges on every backed-up system and will thereby gain full root privileges quickly. IMO that depends on whether you also backup the authentication-related files or not. Exclude them from backup, ensure different root passwords for all boxes, and now you can limit the infiltration. If you're pulling to the backup server, that backup server has to be able to log in to and read all files on the other servers. Including e.g. your swap partition and device files. What if I have each system save a copy of everything to be backed up from its own filesystem in a separate directory and change the ownership of everything in that directory so it can be read by an unprivileged backup user? Then I could have the backup server pull that copy from each system without giving it root access to each system. Can I somehow have the correct ownerships for the backup saved in a separate file for use during a restore? - Grant You could just as well use an NFS share with no_root_squash. It is really more a question of finding the right combination of tools to ensure proper separation of concern for server and client. In fact, I think we are intermixing three distinct problems: 1. (Possible) limitations of rdiff-backup with regard to untrusted backup servers or clients. The limitation is real unfortunately. All backups created by rdiff-backup more than a second ago can be deleted something like this: rdiff-backup --remove-older-than 1s backup@12.34.56.78::/path/to/backup 2. The purely technical question which file transfer protocols protect against write access from backup server to backup client and backup client to older backups on the server. rdiff-backup doesn't provide those sort of protections. Do any file transfer protocols? 3. The more or less organisational question what level of protection backups need and how fast security breaks have to be detected. I think backups should be very well protected and security breaks should not have to be immediately detected. - Grant I think push vs. pull is just a secondary concern with regard to the second question and has practically no relevance to the third one. Regards, Florian Philipp
Re: [gentoo-user] {OT} Are push backups flawed?
Sure, but you've noted that rdiff-backup is insecure if the source box is violated. What you need, then, is a layer of versioning not subject to that vulnerability. ZZ On Nov 14, 2011 9:34 PM, Grant emailgr...@gmail.com wrote: You identified a flaw in the system as you were using it. You're right, those are flaws. However, you can fix those flaws by applying some magic as a sysadmin. That's why several posts in the thread have mentioned versioning your backups in some fashion. I've mentioned lvm a couple times. I thought versioning meant that you could roll back to a previous version. rdiff-backup provides that. I think someone else mentioned pulling the backup target's data to another locale, either via a pull from another server, or via something like a traditional incremental tape backup. So the systems push to the backup server and a second backup server pulls from the first backup server? Should the second backup server use rdiff-backup against the rdiff-backup repository on the first backup server? I think I've read that's not a good idea. What does everybody else do? I feel like the first person to ever attempt secure automated backups. - Grant You're getting the data off the original machines to a remote location, which is good. You identified a way the backed-up data could be tampered with, which is good. You just need to put in another (better) barrier to protect the data from being tampered with, or limit how much data is lost in such an event. ZZ On Nov 14, 2011 8:21 PM, Grant emailgr...@gmail.com wrote: It's out of scope for file transfer protocols; it's a daemon/system-local problem. Attach pre-event or post-event scripts serverside for any special munging or protections you'd like to apply. (Such as triggering an LVM snapshot, for example...) I must be going about this the wrong way. Am I the only one using automated backups? If not, how is it done properly? - Grant And if I pull, none of my backed-up systems are secure because anyone who breaks into the backup server has root read privileges on every backed-up system and will thereby gain full root privileges quickly. IMO that depends on whether you also backup the authentication-related files or not. Exclude them from backup, ensure different root passwords for all boxes, and now you can limit the infiltration. If you're pulling to the backup server, that backup server has to be able to log in to and read all files on the other servers. Including e.g. your swap partition and device files. What if I have each system save a copy of everything to be backed up from its own filesystem in a separate directory and change the ownership of everything in that directory so it can be read by an unprivileged backup user? Then I could have the backup server pull that copy from each system without giving it root access to each system. Can I somehow have the correct ownerships for the backup saved in a separate file for use during a restore? - Grant You could just as well use an NFS share with no_root_squash. It is really more a question of finding the right combination of tools to ensure proper separation of concern for server and client. In fact, I think we are intermixing three distinct problems: 1. (Possible) limitations of rdiff-backup with regard to untrusted backup servers or clients. The limitation is real unfortunately. All backups created by rdiff-backup more than a second ago can be deleted something like this: rdiff-backup --remove-older-than 1s backup@12.34.56.78: :/path/to/backup 2. The purely technical question which file transfer protocols protect against write access from backup server to backup client and backup client to older backups on the server. rdiff-backup doesn't provide those sort of protections. Do any file transfer protocols? 3. The more or less organisational question what level of protection backups need and how fast security breaks have to be detected. I think backups should be very well protected and security breaks should not have to be immediately detected. - Grant I think push vs. pull is just a secondary concern with regard to the second question and has practically no relevance to the third one. Regards, Florian Philipp
Re: [gentoo-user] {OT} Are push backups flawed?
Sure, but you've noted that rdiff-backup is insecure if the source box is violated. What you need, then, is a layer of versioning not subject to that vulnerability. Does it exist? - Grant You identified a flaw in the system as you were using it. You're right, those are flaws. However, you can fix those flaws by applying some magic as a sysadmin. That's why several posts in the thread have mentioned versioning your backups in some fashion. I've mentioned lvm a couple times. I thought versioning meant that you could roll back to a previous version. rdiff-backup provides that. I think someone else mentioned pulling the backup target's data to another locale, either via a pull from another server, or via something like a traditional incremental tape backup. So the systems push to the backup server and a second backup server pulls from the first backup server? Should the second backup server use rdiff-backup against the rdiff-backup repository on the first backup server? I think I've read that's not a good idea. What does everybody else do? I feel like the first person to ever attempt secure automated backups. - Grant You're getting the data off the original machines to a remote location, which is good. You identified a way the backed-up data could be tampered with, which is good. You just need to put in another (better) barrier to protect the data from being tampered with, or limit how much data is lost in such an event. ZZ On Nov 14, 2011 8:21 PM, Grant emailgr...@gmail.com wrote: It's out of scope for file transfer protocols; it's a daemon/system-local problem. Attach pre-event or post-event scripts serverside for any special munging or protections you'd like to apply. (Such as triggering an LVM snapshot, for example...) I must be going about this the wrong way. Am I the only one using automated backups? If not, how is it done properly? - Grant And if I pull, none of my backed-up systems are secure because anyone who breaks into the backup server has root read privileges on every backed-up system and will thereby gain full root privileges quickly. IMO that depends on whether you also backup the authentication-related files or not. Exclude them from backup, ensure different root passwords for all boxes, and now you can limit the infiltration. If you're pulling to the backup server, that backup server has to be able to log in to and read all files on the other servers. Including e.g. your swap partition and device files. What if I have each system save a copy of everything to be backed up from its own filesystem in a separate directory and change the ownership of everything in that directory so it can be read by an unprivileged backup user? Then I could have the backup server pull that copy from each system without giving it root access to each system. Can I somehow have the correct ownerships for the backup saved in a separate file for use during a restore? - Grant You could just as well use an NFS share with no_root_squash. It is really more a question of finding the right combination of tools to ensure proper separation of concern for server and client. In fact, I think we are intermixing three distinct problems: 1. (Possible) limitations of rdiff-backup with regard to untrusted backup servers or clients. The limitation is real unfortunately. All backups created by rdiff-backup more than a second ago can be deleted something like this: rdiff-backup --remove-older-than 1s backup@12.34.56.78::/path/to/backup 2. The purely technical question which file transfer protocols protect against write access from backup server to backup client and backup client to older backups on the server. rdiff-backup doesn't provide those sort of protections. Do any file transfer protocols? 3. The more or less organisational question what level of protection backups need and how fast security breaks have to be detected. I think backups should be very well protected and security breaks should not have to be immediately detected. - Grant I think push vs. pull is just a secondary concern with regard to the second question and has practically no relevance to the third one. Regards, Florian Philipp
Re: [gentoo-user] {OT} Are push backups flawed?
On Nov 15, 2011 9:49 AM, Grant emailgr...@gmail.com wrote:
Sure, but you've noted that rdiff-backup is insecure if the source box
is
violated. What you need, then, is a layer of versioning not subject to
that
vulnerability.
Does it exist?
Quick and dirty:
TARGDIR=/home/versions/$(date +%Y%m%d%H%M%S)
find $BACKUPDIR -exec ln {} $TARGDIR/{} \; # make hard links; they survive
unlinking of the originals
Disclaimer: Not tested. Use it at your own risk.
Rgds,
Re: [gentoo-user] {OT} Are push backups flawed?
On Tue, November 15, 2011 3:32 am, Grant wrote: You identified a flaw in the system as you were using it. You're right, those are flaws. However, you can fix those flaws by applying some magic as a sysadmin. That's why several posts in the thread have mentioned versioning your backups in some fashion. I've mentioned lvm a couple times. I thought versioning meant that you could roll back to a previous version. rdiff-backup provides that. It's part of it. I think someone else mentioned pulling the backup target's data to another locale, either via a pull from another server, or via something like a traditional incremental tape backup. So the systems push to the backup server and a second backup server pulls from the first backup server? Should the second backup server use rdiff-backup against the rdiff-backup repository on the first backup server? I think I've read that's not a good idea. Not sure, I don't use rdiff-backup. Am considering it for the desktops once the new server is in place. What does everybody else do? I feel like the first person to ever attempt secure automated backups. For more secure backups, you could use backup-utilities that support incremental backups. dar springs to mind. So do larger automated systems. As my servers are all virtual machines running on Xen, I configured pull style backups. For the desktops, I am planning the following: rdiff-backup (or similar) to push backups from the desktops to the server. Adding hardlinks as already suggested for simple versioning. The backup-script in the desktop will do 2 things: 1) rdiff-backup 2) instruct the backup-server to create the hardlinks with versioning Then, at regular intervals, this will be backed up by pull from the Host-domain on the server. I don't see any chance to kill all my backups as the data will remain, even when deleting the backup-directory of a desktop. -- Joost
Re: [gentoo-user] {OT} Are push backups flawed?
And if I pull, none of my backed-up systems are secure because anyone who breaks into the backup server has root read privileges on every backed-up system and will thereby gain full root privileges quickly. IMO that depends on whether you also backup the authentication-related files or not. Exclude them from backup, ensure different root passwords for all boxes, and now you can limit the infiltration. If you're pulling to the backup server, that backup server has to be able to log in to and read all files on the other servers. Including e.g. your swap partition and device files. What if I have each system save a copy of everything to be backed up from its own filesystem in a separate directory and change the ownership of everything in that directory so it can be read by an unprivileged backup user? Then I could have the backup server pull that copy from each system without giving it root access to each system. Can I somehow have the correct ownerships for the backup saved in a separate file for use during a restore? - Grant
Re: [gentoo-user] {OT} Are push backups flawed?
Am 13.11.2011 19:03, schrieb Grant: And if I pull, none of my backed-up systems are secure because anyone who breaks into the backup server has root read privileges on every backed-up system and will thereby gain full root privileges quickly. IMO that depends on whether you also backup the authentication-related files or not. Exclude them from backup, ensure different root passwords for all boxes, and now you can limit the infiltration. If you're pulling to the backup server, that backup server has to be able to log in to and read all files on the other servers. Including e.g. your swap partition and device files. What if I have each system save a copy of everything to be backed up from its own filesystem in a separate directory and change the ownership of everything in that directory so it can be read by an unprivileged backup user? Then I could have the backup server pull that copy from each system without giving it root access to each system. Can I somehow have the correct ownerships for the backup saved in a separate file for use during a restore? - Grant You could just as well use an NFS share with no_root_squash. It is really more a question of finding the right combination of tools to ensure proper separation of concern for server and client. In fact, I think we are intermixing three distinct problems: 1. (Possible) limitations of rdiff-backup with regard to untrusted backup servers or clients. 2. The purely technical question which file transfer protocols protect against write access from backup server to backup client and backup client to older backups on the server. 3. The more or less organisational question what level of protection backups need and how fast security breaks have to be detected. I think push vs. pull is just a secondary concern with regard to the second question and has practically no relevance to the third one. Regards, Florian Philipp signature.asc Description: OpenPGP digital signature
Re: [gentoo-user] {OT} Are push backups flawed?
On 11/13/11 13:03, Grant wrote: And if I pull, none of my backed-up systems are secure because anyone who breaks into the backup server has root read privileges on every backed-up system and will thereby gain full root privileges quickly. IMO that depends on whether you also backup the authentication-related files or not. Exclude them from backup, ensure different root passwords for all boxes, and now you can limit the infiltration. If you're pulling to the backup server, that backup server has to be able to log in to and read all files on the other servers. Including e.g. your swap partition and device files. What if I have each system save a copy of everything to be backed up from its own filesystem in a separate directory and change the ownership of everything in that directory so it can be read by an unprivileged backup user? You've just reinvented the push backup =) If separate-directory is on the same server, an attacker can log in and overwrite all of your files with zeros. Those zeros will be pulled to the backup server, destroying your backups. If separate-directory is on the backup server...
Re: [gentoo-user] {OT} Are push backups flawed?
On 11/13/11 13:03, Grant wrote: Then I could have the backup server pull that copy from each system without giving it root access to each system. Can I somehow have the correct ownerships for the backup saved in a separate file for use during a restore? If you're intent on making a two-stage pull work; you can do it by creating a 'backups' user on your servers, and then using filesystem ACLs to grant backups+r to every file/directory you want to back up. That way, an attacker on the backup server can't decide to peruse the rest of your stuff. The easiest method, though, is to just add a third stage. Either move the backups on the backup server to another directory after the backup job completes, or sync/burn/whatever them off-site. In this case the backup server can't access anything you don't give it, and the individual servers can't trash their backed-up data.
[gentoo-user] {OT} Are push backups flawed?
A little while ago I set up an automated backup system to back up the data from 3 machines to a backup server. I decided to use a push-style layout where the 3 machines push their data to the backup server. Public SSH keys for the 3 machines are stored on the backup server and restricted to the rdiff-backup command. Each of the 3 machines pushes their data to the backup server as a different user and the top directory of each backup is chmod 700 to prevent any of the 3 machines from reading or writing a backup from another machine. I've run into a problem with this layout that I can't seem to solve, and I'm wondering if I should switch to a pull-style layout where the backup server pulls data from each of the 3 machines. The problem with my current push-style layout is that if one of the 3 machines is compromised, the attacker can delete or alter the backup of the compromised machine on the backup server. I can rsync the backups from the backup server to another machine, but if the backups are deleted or altered on the backup server, the rsync'ed copy on the next machine will also be deleted or altered. If I run a pull-style layout and the backup server is compromised, the attacker would have root read access to each of the 3 machines, but the attacker would already have access to backups from each of the 3 machines stored on the backup server itself so that's not really an issue. I would also have the added inconvenience of using openvpn or ssh -R for my laptop so the backup server can pull from it through any router. What do you think guys? Are push-style backups flawed and unacceptable? - Grant
Re: [gentoo-user] {OT} Are push backups flawed?
On Fri, Nov 11, 2011 at 12:55 PM, Grant emailgr...@gmail.com wrote: [snip] The problem with my current push-style layout is that if one of the 3 machines is compromised, the attacker can delete or alter the backup of the compromised machine on the backup server. I can rsync the backups from the backup server to another machine, but if the backups are deleted or altered on the backup server, the rsync'ed copy on the next machine will also be deleted or altered. As a final stage in your backup, could you trigger a 'pull'-style backup copying the data image to a more secure area? How about setting your backup target on top of lvm, and snapshotting? Some mechanism could be employed so that the snapshot command is run by a more restricted user, and done so after, e.g. a certain amount of idle time in the backup target directory If I run a pull-style layout and the backup server is compromised, the attacker would have root read access to each of the 3 machines, but the attacker would already have access to backups from each of the 3 machines stored on the backup server itself so that's not really an issue. I would also have the added inconvenience of using openvpn or ssh -R for my laptop so the backup server can pull from it through any router. Check out freenet6. I use it so that my laptop has a static, global IP address whether it's on my home network or not. It's quite nice. IPv6 in various applications also solves my other direct-access needs. What do you think guys? Are push-style backups flawed and unacceptable? I imagine you might still want to 'pull' from your backup server; if someone gets a key that allows them to manipulate the behavior of a local process that shouldn't normally be manipulated, your vulnerability surface goes up. -- :wq
Re: [gentoo-user] {OT} Are push backups flawed?
On Nov 12, 2011 12:58 AM, Grant emailgr...@gmail.com wrote: A little while ago I set up an automated backup system to back up the data from 3 machines to a backup server. I decided to use a push-style layout where the 3 machines push their data to the backup server. Public SSH keys for the 3 machines are stored on the backup server and restricted to the rdiff-backup command. Each of the 3 machines pushes their data to the backup server as a different user and the top directory of each backup is chmod 700 to prevent any of the 3 machines from reading or writing a backup from another machine. I've run into a problem with this layout that I can't seem to solve, and I'm wondering if I should switch to a pull-style layout where the backup server pulls data from each of the 3 machines. The problem with my current push-style layout is that if one of the 3 machines is compromised, the attacker can delete or alter the backup of the compromised machine on the backup server. I can rsync the backups from the backup server to another machine, but if the backups are deleted or altered on the backup server, the rsync'ed copy on the next machine will also be deleted or altered. If I run a pull-style layout and the backup server is compromised, the attacker would have root read access to each of the 3 machines, but the attacker would already have access to backups from each of the 3 machines stored on the backup server itself so that's not really an issue. I would also have the added inconvenience of using openvpn or ssh -R for my laptop so the backup server can pull from it through any router. What do you think guys? Are push-style backups flawed and unacceptable? No, it's not flawed, as long as the implementation is right: versioning and deduplication. With versioning, an attacker (or infiltrator, in this matter) might try to taint the backup, but all she can do is just push a new version to the server. You can recover your data by reverting to a prior version. The deduplication part is only to save storage space. It's less necessary if you have a robust versioning system that can categorize each push as either canonical/perpetual/permanent or ephemeral/temporary. The system can just discard old ephemeral pushes when storage becomes critical. Rgds,
Re: [gentoo-user] {OT} Are push backups flawed?
[snip] The problem with my current push-style layout is that if one of the 3 machines is compromised, the attacker can delete or alter the backup of the compromised machine on the backup server. I can rsync the backups from the backup server to another machine, but if the backups are deleted or altered on the backup server, the rsync'ed copy on the next machine will also be deleted or altered. As a final stage in your backup, could you trigger a 'pull'-style backup copying the data image to a more secure area? How about setting Even if I pull a copy of the backup to a separate machine from the backup server, it will pull an altered copy if an attacker compromises one of the systems being backed up and alters that system's backup on the backup server. Am I missing something? - Grant your backup target on top of lvm, and snapshotting? Some mechanism could be employed so that the snapshot command is run by a more restricted user, and done so after, e.g. a certain amount of idle time in the backup target directory If I run a pull-style layout and the backup server is compromised, the attacker would have root read access to each of the 3 machines, but the attacker would already have access to backups from each of the 3 machines stored on the backup server itself so that's not really an issue. I would also have the added inconvenience of using openvpn or ssh -R for my laptop so the backup server can pull from it through any router. Check out freenet6. I use it so that my laptop has a static, global IP address whether it's on my home network or not. It's quite nice. IPv6 in various applications also solves my other direct-access needs. What do you think guys? Are push-style backups flawed and unacceptable? I imagine you might still want to 'pull' from your backup server; if someone gets a key that allows them to manipulate the behavior of a local process that shouldn't normally be manipulated, your vulnerability surface goes up. -- :wq
Re: [gentoo-user] {OT} Are push backups flawed?
On Fri, Nov 11, 2011 at 1:27 PM, Grant emailgr...@gmail.com wrote: [snip] The problem with my current push-style layout is that if one of the 3 machines is compromised, the attacker can delete or alter the backup of the compromised machine on the backup server. I can rsync the backups from the backup server to another machine, but if the backups are deleted or altered on the backup server, the rsync'ed copy on the next machine will also be deleted or altered. As a final stage in your backup, could you trigger a 'pull'-style backup copying the data image to a more secure area? How about setting Even if I pull a copy of the backup to a separate machine from the backup server, it will pull an altered copy if an attacker compromises one of the systems being backed up and alters that system's backup on the backup server. Am I missing something? If you're not applying any kind of versioning, it doesn't matter if you're pushing or pulling; your backup will eventually be overwritten by a backup of a hacked system unless you catch and respond as soon as the original invasion happens. So it sounds like the scenario you fear isn't tied to the mechanism you're reconsidering. -- :wq
Re: [gentoo-user] {OT} Are push backups flawed?
A little while ago I set up an automated backup system to back up the data from 3 machines to a backup server. I decided to use a push-style layout where the 3 machines push their data to the backup server. Public SSH keys for the 3 machines are stored on the backup server and restricted to the rdiff-backup command. Each of the 3 machines pushes their data to the backup server as a different user and the top directory of each backup is chmod 700 to prevent any of the 3 machines from reading or writing a backup from another machine. I've run into a problem with this layout that I can't seem to solve, and I'm wondering if I should switch to a pull-style layout where the backup server pulls data from each of the 3 machines. The problem with my current push-style layout is that if one of the 3 machines is compromised, the attacker can delete or alter the backup of the compromised machine on the backup server. I can rsync the backups from the backup server to another machine, but if the backups are deleted or altered on the backup server, the rsync'ed copy on the next machine will also be deleted or altered. If I run a pull-style layout and the backup server is compromised, the attacker would have root read access to each of the 3 machines, but the attacker would already have access to backups from each of the 3 machines stored on the backup server itself so that's not really an issue. I would also have the added inconvenience of using openvpn or ssh -R for my laptop so the backup server can pull from it through any router. What do you think guys? Are push-style backups flawed and unacceptable? No, it's not flawed, as long as the implementation is right: versioning and deduplication. With versioning, an attacker (or infiltrator, in this matter) might try to taint the backup, but all she can do is just push a new version to the server. You can recover your data by reverting to a prior version. Is that true? Wouldn't the infiltrator be able to craft some sort of rdiff-backup command that deletes the entire backup? I can't come up with such a command myself, but I thought I was essentially giving full read/write access of a system's backup to an infiltrator by putting that system's public key on the backup server. I do restrict the key like command=rdiff-backup --server but I didn't expect that to completely prevent the backup from being wiped out. Does it? - Grant The deduplication part is only to save storage space. It's less necessary if you have a robust versioning system that can categorize each push as either canonical/perpetual/permanent or ephemeral/temporary. The system can just discard old ephemeral pushes when storage becomes critical.
Re: [gentoo-user] {OT} Are push backups flawed?
On Nov 12, 2011 1:39 AM, Grant emailgr...@gmail.com wrote: A little while ago I set up an automated backup system to back up the data from 3 machines to a backup server. I decided to use a push-style layout where the 3 machines push their data to the backup server. Public SSH keys for the 3 machines are stored on the backup server and restricted to the rdiff-backup command. Each of the 3 machines pushes their data to the backup server as a different user and the top directory of each backup is chmod 700 to prevent any of the 3 machines from reading or writing a backup from another machine. I've run into a problem with this layout that I can't seem to solve, and I'm wondering if I should switch to a pull-style layout where the backup server pulls data from each of the 3 machines. The problem with my current push-style layout is that if one of the 3 machines is compromised, the attacker can delete or alter the backup of the compromised machine on the backup server. I can rsync the backups from the backup server to another machine, but if the backups are deleted or altered on the backup server, the rsync'ed copy on the next machine will also be deleted or altered. If I run a pull-style layout and the backup server is compromised, the attacker would have root read access to each of the 3 machines, but the attacker would already have access to backups from each of the 3 machines stored on the backup server itself so that's not really an issue. I would also have the added inconvenience of using openvpn or ssh -R for my laptop so the backup server can pull from it through any router. What do you think guys? Are push-style backups flawed and unacceptable? No, it's not flawed, as long as the implementation is right: versioning and deduplication. With versioning, an attacker (or infiltrator, in this matter) might try to taint the backup, but all she can do is just push a new version to the server. You can recover your data by reverting to a prior version. Is that true? Wouldn't the infiltrator be able to craft some sort of rdiff-backup command that deletes the entire backup? I can't come up with such a command myself, but I thought I was essentially giving full read/write access of a system's backup to an infiltrator by putting that system's public key on the backup server. I do restrict the key like command=rdiff-backup --server but I didn't expect that to completely prevent the backup from being wiped out. Does it? - Grant The deduplication part is only to save storage space. It's less necessary if you have a robust versioning system that can categorize each push as either canonical/perpetual/permanent or ephemeral/temporary. The system can just discard old ephemeral pushes when storage becomes critical. Just an illustration: My employer will soon do a PoC/Live Demo of this product: http://www.atempo.com/products/liveBackup/features.asp Only an 'agent' lives inside the employee's workstation. It pushes all writes to certain folders to the server, and able to request 'reverts' to their local copy, but the server's archives are immutable. Unfortunately, said product only supports Windows and Macs. I'm still on the lookout for something similar for Linux. (For pure text files, a git/mercurial server would be enough, though.) Rgds,
Re: [gentoo-user] {OT} Are push backups flawed?
Am 11.11.2011 19:56, schrieb Pandu Poluan: On Nov 12, 2011 1:39 AM, Grant emailgr...@gmail.com mailto:emailgr...@gmail.com wrote: A little while ago I set up an automated backup system to back up the data from 3 machines to a backup server. I decided to use a push-style layout where the 3 machines push their data to the backup server. Public SSH keys for the 3 machines are stored on the backup server and restricted to the rdiff-backup command. Each of the 3 machines pushes their data to the backup server as a different user and the top directory of each backup is chmod 700 to prevent any of the 3 machines from reading or writing a backup from another machine. I've run into a problem with this layout that I can't seem to solve, and I'm wondering if I should switch to a pull-style layout where the backup server pulls data from each of the 3 machines. The problem with my current push-style layout is that if one of the 3 machines is compromised, the attacker can delete or alter the backup of the compromised machine on the backup server. I can rsync the backups from the backup server to another machine, but if the backups are deleted or altered on the backup server, the rsync'ed copy on the next machine will also be deleted or altered. If I run a pull-style layout and the backup server is compromised, the attacker would have root read access to each of the 3 machines, but the attacker would already have access to backups from each of the 3 machines stored on the backup server itself so that's not really an issue. I would also have the added inconvenience of using openvpn or ssh -R for my laptop so the backup server can pull from it through any router. What do you think guys? Are push-style backups flawed and unacceptable? No, it's not flawed, as long as the implementation is right: versioning and deduplication. With versioning, an attacker (or infiltrator, in this matter) might try to taint the backup, but all she can do is just push a new version to the server. You can recover your data by reverting to a prior version. Is that true? Wouldn't the infiltrator be able to craft some sort of rdiff-backup command that deletes the entire backup? I can't come up with such a command myself, but I thought I was essentially giving full read/write access of a system's backup to an infiltrator by putting that system's public key on the backup server. I do restrict the key like command=rdiff-backup --server but I didn't expect that to completely prevent the backup from being wiped out. Does it? - Grant The deduplication part is only to save storage space. It's less necessary if you have a robust versioning system that can categorize each push as either canonical/perpetual/permanent or ephemeral/temporary. The system can just discard old ephemeral pushes when storage becomes critical. Just an illustration: My employer will soon do a PoC/Live Demo of this product: http://www.atempo.com/products/liveBackup/features.asp Only an 'agent' lives inside the employee's workstation. It pushes all writes to certain folders to the server, and able to request 'reverts' to their local copy, but the server's archives are immutable. Unfortunately, said product only supports Windows and Macs. I'm still on the lookout for something similar for Linux. (For pure text files, a git/mercurial server would be enough, though.) Rgds, Isn't Bacula something like this? http://www.bacula.org/en/dev-manual/main/main/What_is_Bacula.html#SECTION0022 Hint: File server actually is the client that is backed up. signature.asc Description: OpenPGP digital signature
Re: [gentoo-user] {OT} Are push backups flawed?
On 11/11/2011 12:55 PM, Grant wrote: The problem with my current push-style layout is that if one of the 3 machines is compromised, the attacker can delete or alter the backup of the compromised machine on the backup server. I can rsync the backups from the backup server to another machine, but if the backups are deleted or altered on the backup server, the rsync'ed copy on the next machine will also be deleted or altered. If I run a pull-style layout and the backup server is compromised, the attacker would have root read access to each of the 3 machines, but the attacker would already have access to backups from each of the 3 machines stored on the backup server itself so that's not really an issue. I would also have the added inconvenience of using openvpn or ssh -R for my laptop so the backup server can pull from it through any router. If an attacker can read the entire filesystem, he'll gain full root privileges quickly.
Re: [gentoo-user] {OT} Are push backups flawed?
The problem with my current push-style layout is that if one of the 3 machines is compromised, the attacker can delete or alter the backup of the compromised machine on the backup server. I can rsync the backups from the backup server to another machine, but if the backups are deleted or altered on the backup server, the rsync'ed copy on the next machine will also be deleted or altered. If I run a pull-style layout and the backup server is compromised, the attacker would have root read access to each of the 3 machines, but the attacker would already have access to backups from each of the 3 machines stored on the backup server itself so that's not really an issue. I would also have the added inconvenience of using openvpn or ssh -R for my laptop so the backup server can pull from it through any router. If an attacker can read the entire filesystem, he'll gain full root privileges quickly. So if I push, I don't really have backups because anyone who breaks into the backed-up system can delete all of its backups like this: rdiff-backup --remove-older-than 1s backup@12.34.56.78::/path/to/backup And if I pull, none of my backed-up systems are secure because anyone who breaks into the backup server has root read privileges on every backed-up system and will thereby gain full root privileges quickly. - Grant
Re: [gentoo-user] {OT} Are push backups flawed?
On Nov 12, 2011 9:29 AM, Grant emailgr...@gmail.com wrote: The problem with my current push-style layout is that if one of the 3 machines is compromised, the attacker can delete or alter the backup of the compromised machine on the backup server. I can rsync the backups from the backup server to another machine, but if the backups are deleted or altered on the backup server, the rsync'ed copy on the next machine will also be deleted or altered. If I run a pull-style layout and the backup server is compromised, the attacker would have root read access to each of the 3 machines, but the attacker would already have access to backups from each of the 3 machines stored on the backup server itself so that's not really an issue. I would also have the added inconvenience of using openvpn or ssh -R for my laptop so the backup server can pull from it through any router. If an attacker can read the entire filesystem, he'll gain full root privileges quickly. So if I push, I don't really have backups because anyone who breaks into the backed-up system can delete all of its backups like this: rdiff-backup --remove-older-than 1s backup@12.34.56.78::/path/to/backup Write a daemon that immediately create hardlinks of the backed-up files in a separate folder. Thus, even if rdiff decides to unlink everything, your data are safe thanks to the nature of hardlinks. Optionally, have the same daemon tarball the files (via the hardlinks) if you deem the revision 'permanent'. And if I pull, none of my backed-up systems are secure because anyone who breaks into the backup server has root read privileges on every backed-up system and will thereby gain full root privileges quickly. IMO that depends on whether you also backup the authentication-related files or not. Exclude them from backup, ensure different root passwords for all boxes, and now you can limit the infiltration. Rgds,
Re: [gentoo-user] {OT} Are push backups flawed?
On Nov 12, 2011 2:17 AM, Florian Philipp li...@binarywings.net wrote: Just an illustration: My employer will soon do a PoC/Live Demo of this product: http://www.atempo.com/products/liveBackup/features.asp Only an 'agent' lives inside the employee's workstation. It pushes all writes to certain folders to the server, and able to request 'reverts' to their local copy, but the server's archives are immutable. Unfortunately, said product only supports Windows and Macs. I'm still on the lookout for something similar for Linux. (For pure text files, a git/mercurial server would be enough, though.) Rgds, Isn't Bacula something like this? http://www.bacula.org/en/dev-manual/main/main/What_is_Bacula.html#SECTION0022 Hint: File server actually is the client that is backed up. Thanks! I knew someone has created something similar for Linux, but the name escaped my mind :-) Rgds,
Re: [gentoo-user] {OT} Are push backups flawed?
On 11/11/2011 09:22 PM, Grant wrote: So if I push, I don't really have backups because anyone who breaks into the backed-up system can delete all of its backups like this: rdiff-backup --remove-older-than 1s backup@12.34.56.78::/path/to/backup And if I pull, none of my backed-up systems are secure because anyone who breaks into the backup server has root read privileges on every backed-up system and will thereby gain full root privileges quickly. It's a false dichotomy[1], but sums up the trade-off between those two options well enough. The last hacker who tried to delete everything on my system was a 5.25in floppy. So, I'm biased towards the other case. [1] Third option: choose push or pull, and ALSO make off-site read-only backups of the backup server every once in a while.
Re: [gentoo-user] {OT} Are push backups flawed?
On 11/11/2011 10:20 PM, Pandu Poluan wrote: And if I pull, none of my backed-up systems are secure because anyone who breaks into the backup server has root read privileges on every backed-up system and will thereby gain full root privileges quickly. IMO that depends on whether you also backup the authentication-related files or not. Exclude them from backup, ensure different root passwords for all boxes, and now you can limit the infiltration. If you're pulling to the backup server, that backup server has to be able to log in to and read all files on the other servers. Including e.g. your swap partition and device files.
Re: [gentoo-user] {OT} Are push backups flawed?
On Nov 12, 2011 11:23 AM, Michael Orlitzky mich...@orlitzky.com wrote: On 11/11/2011 10:20 PM, Pandu Poluan wrote: And if I pull, none of my backed-up systems are secure because anyone who breaks into the backup server has root read privileges on every backed-up system and will thereby gain full root privileges quickly. IMO that depends on whether you also backup the authentication-related files or not. Exclude them from backup, ensure different root passwords for all boxes, and now you can limit the infiltration. If you're pulling to the backup server, that backup server has to be able to log in to and read all files on the other servers. Including e.g. your swap partition and device files. Again, that's a matter of implementation. If the server doesn't access the client's filesystem directly but via an agent (Bacula does this, for instance), the server's access will be limited to what the agent provides. Rgds,
