Re: [gentoo-user] {OT} Are "push" backups flawed?

[Michael Orlitzky]

On 11/14/11 20:54, Grant wrote:
>>
>> If you're intent on making a two-stage pull work; you can do it by
>> creating a 'backups' user on your servers, and then using filesystem
>> ACLs to grant backups+r to every file/directory you want to back up.
>> That way, an attacker on the backup server can't decide to peruse the
>> rest of your stuff.
> 
> I like that.  So use ACLs to grant access to the backups instead of
> using ownership/permissions so that the ownership/permissions stay
> intact.  I've never used ACLs.  Do they "override"
> ownership/permissions?  In other words, if the ACL specifies backups+r
> to a file owned by root that is chmod 700, "backups" can read it
> anyway?

Yup, they work like Windows ACLs if you've used those. You can grant one
user read permission without affecting anything else.

The '700' mode doesn't really make sense anymore after you apply an
ACL.. the whole permissions-as-bits concept gets highly convoluted[1]
but if you just want to add read access for one user it's easy.

You can use setfacl to add permissions, and double-check with getfacl
that they do what you think they do. The examples in `man setfacl` are
pretty easy to understand.


>> The easiest method, though, is to just add a third stage. Either move
>> the backups on the backup server to another directory after the backup
>> job completes, or sync/burn/whatever them off-site. In this case the
>> backup server can't access anything you don't give it, and the
>> individual servers can't trash their backed-up data.
> 
> I don't see how that could work in an automated fashion.  Could you
> give me an example?

We do push backups to one server, backup1, every night. Then, every day,
backup1 syncs to another server, backup2. The individuals servers have
no access to backup2, and it's physically separate from backup1.

I make physical, removable, backups of backup2 every once in a while,
but not as often as I should.


[1] http://www.suse.de/~agruen/acl/linux-acls/online/

Re: [gentoo-user] {OT} Are "push" backups flawed?

[J. Roeleveld]

On Tue, November 15, 2011 3:32 am, Grant wrote:
>> You identified a flaw in the system as you were using it. You're right,
>> those are flaws. However, you can " fix" those flaws by applying some
>> magic
>> as a sysadmin. That's why several posts in the thread have mentioned
>> versioning your backups in some fashion. I've mentioned lvm a couple
>> times.
>
> I thought versioning meant that you could roll back to a previous
> version.  rdiff-backup provides that.

It's part of it.

>> I think someone else mentioned pulling the backup target's data to
>> another
>> locale, either via a pull from another server, or via something like a
>> traditional incremental tape backup.
>
> So the systems push to the backup server and a second backup server
> pulls from the first backup server?  Should the second backup server
> use rdiff-backup against the rdiff-backup repository on the first
> backup server?  I think I've read that's not a good idea.

Not sure, I don't use rdiff-backup. Am considering it for the desktops
once the new server is in place.

> What does everybody else do?  I feel like the first person to ever
> attempt secure automated backups.

For more secure backups, you could use backup-utilities that support
incremental backups.
"dar" springs to mind. So do larger automated systems.

As my servers are all virtual machines running on Xen, I configured "pull"
style backups.

For the desktops, I am planning the following:
"rdiff-backup" (or similar) to push backups from the desktops to the
server. Adding hardlinks as already suggested for simple versioning.
The backup-script in the desktop will do 2 things:
1) rdiff-backup
2) instruct the backup-server to create the hardlinks with versioning

Then, at regular intervals, this will be backed up by "pull" from the
Host-domain on the server.

I don't see any chance to kill all my backups as the data will remain,
even when deleting the backup-directory of a desktop.

--
Joost

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Pandu Poluan]

On Nov 15, 2011 9:49 AM, "Grant"  wrote:
>
> > Sure, but you've noted that rdiff-backup is insecure if the source box
is
> > violated. What you need, then, is a layer of versioning not subject to
that
> > vulnerability.
>
> Does it exist?
>

Quick and dirty:

TARGDIR=/home/versions/$(date +"%Y%m%d%H%M%S")
find $BACKUPDIR -exec ln {} $TARGDIR/{} \;  # make hard links; they survive
unlinking of the originals

Disclaimer: Not tested. Use it at your own risk.

Rgds,

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Grant]

> Sure, but you've noted that rdiff-backup is insecure if the source box is
> violated. What you need, then, is a layer of versioning not subject to that
> vulnerability.

Does it exist?

- Grant


>> > You identified a flaw in the system as you were using it. You're right,
>> > those are flaws. However, you can " fix" those flaws by applying some
>> > magic
>> > as a sysadmin. That's why several posts in the thread have mentioned
>> > versioning your backups in some fashion. I've mentioned lvm a couple
>> > times.
>>
>> I thought versioning meant that you could roll back to a previous
>> version.  rdiff-backup provides that.
>>
>> > I think someone else mentioned pulling the backup target's data to
>> > another
>> > locale, either via a pull from another server, or via something like a
>> > traditional incremental tape backup.
>>
>> So the systems push to the backup server and a second backup server
>> pulls from the first backup server?  Should the second backup server
>> use rdiff-backup against the rdiff-backup repository on the first
>> backup server?  I think I've read that's not a good idea.
>>
>> What does everybody else do?  I feel like the first person to ever
>> attempt secure automated backups.
>>
>> - Grant
>>
>>
>> > You're getting the data off the original machines to a remote location,
>> > which is good. You identified a way the backed-up data could be tampered
>> > with, which is good. You just need to put in another (better) barrier to
>> > protect the data from being tampered with, or limit how much data is
>> > lost in
>> > such an event.
>> >
>> > ZZ
>> >
>> > On Nov 14, 2011 8:21 PM, "Grant"  wrote:
>> >>
>> >> > It's out of scope for file transfer protocols; it's a
>> >> > daemon/system-local
>> >> > problem.  Attach pre-event or post-event scripts serverside for any
>> >> > special
>> >> > munging or protections you'd like to apply. (Such as triggering an
>> >> > LVM
>> >> > snapshot, for example...)
>> >>
>> >> I must be going about this the wrong way.  Am I the only one using
>> >> automated backups?  If not, how is it done properly?
>> >>
>> >> - Grant
>> >>
>> >>
>> >> >> > And if I pull, none of my backed-up systems are secure because
>> >> >> > anyone
>> >> >> > who breaks into the backup server has root read privileges on
>> >> >> > every
>> >> >> > backed-up system and will thereby "gain full root privileges
>> >> >> > quickly."
>> >> >> 
>> >> >>  IMO that depends on whether you also backup the
>> >> >>  authentication-related
>> >> >>  files or not. Exclude them from backup, ensure different root
>> >> >>  passwords
>> >> >>  for all boxes, and now you can limit the infiltration.
>> >> >> >>>
>> >> >> >>> If you're pulling to the backup server, that backup server has
>> >> >> >>> to
>> >> >> >>> be
>> >> >> >>> able to log in to and read all files on the other servers.
>> >> >> >>> Including
>> >> >> >>> e.g. your swap partition and device files.
>> >> >> >>
>> >> >> >> What if I have each system save a copy of everything to be backed
>> >> >> >> up
>> >> >> >> from its own filesystem in a separate directory and change the
>> >> >> >> ownership of everything in that directory so it can be read by an
>> >> >> >> unprivileged backup user?  Then I could have the backup server
>> >> >> >> pull
>> >> >> >> that copy from each system without giving it root access to each
>> >> >> >> system.  Can I somehow have the correct ownerships for the backup
>> >> >> >> saved in a separate file for use during a restore?
>> >> >> >>
>> >> >> >> - Grant
>> >> >> >>
>> >> >> >
>> >> >> > You could just as well use an NFS share with no_root_squash. It is
>> >> >> > really more a question of finding the right combination of tools
>> >> >> > to
>> >> >> > ensure proper separation of concern for server and client.
>> >> >> >
>> >> >> > In fact, I think we are intermixing three distinct problems:
>> >> >> > 1. (Possible) limitations of rdiff-backup with regard to untrusted
>> >> >> > backup servers or clients.
>> >> >>
>> >> >> The limitation is real unfortunately.  All backups created by
>> >> >> rdiff-backup more than a second ago can be deleted something like
>> >> >> this:
>> >> >>
>> >> >> rdiff-backup --remove-older-than 1s
>> >> >> backup@12.34.56.78::/path/to/backup
>> >> >>
>> >> >> > 2. The purely technical question which file transfer protocols
>> >> >> > protect
>> >> >> > against write access from backup server to backup client and
>> >> >> > backup
>> >> >> > client to older backups on the server.
>> >> >>
>> >> >> rdiff-backup doesn't provide those sort of protections.  Do any file
>> >> >> transfer protocols?
>> >> >>
>> >> >> > 3. The more or less organisational question what level of
>> >> >> > protection
>> >> >> > backups need and how fast security breaks have to be detected.
>> >> >>
>> >> >> I think backups should be very well protected and security breaks
>> >> >> should not have to be immediately detected.
>> >> >>
>> >> >> -

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Michael Mol]

Sure, but you've noted that rdiff-backup is insecure if the source box is
violated. What you need, then, is a layer of versioning not subject to that
vulnerability.

ZZ
On Nov 14, 2011 9:34 PM, "Grant"  wrote:

> > You identified a flaw in the system as you were using it. You're right,
> > those are flaws. However, you can " fix" those flaws by applying some
> magic
> > as a sysadmin. That's why several posts in the thread have mentioned
> > versioning your backups in some fashion. I've mentioned lvm a couple
> times.
>
> I thought versioning meant that you could roll back to a previous
> version.  rdiff-backup provides that.
>
> > I think someone else mentioned pulling the backup target's data to
> another
> > locale, either via a pull from another server, or via something like a
> > traditional incremental tape backup.
>
> So the systems push to the backup server and a second backup server
> pulls from the first backup server?  Should the second backup server
> use rdiff-backup against the rdiff-backup repository on the first
> backup server?  I think I've read that's not a good idea.
>
> What does everybody else do?  I feel like the first person to ever
> attempt secure automated backups.
>
> - Grant
>
>
> > You're getting the data off the original machines to a remote location,
> > which is good. You identified a way the backed-up data could be tampered
> > with, which is good. You just need to put in another (better) barrier to
> > protect the data from being tampered with, or limit how much data is
> lost in
> > such an event.
> >
> > ZZ
> >
> > On Nov 14, 2011 8:21 PM, "Grant"  wrote:
> >>
> >> > It's out of scope for file transfer protocols; it's a
> >> > daemon/system-local
> >> > problem.  Attach pre-event or post-event scripts serverside for any
> >> > special
> >> > munging or protections you'd like to apply. (Such as triggering an LVM
> >> > snapshot, for example...)
> >>
> >> I must be going about this the wrong way.  Am I the only one using
> >> automated backups?  If not, how is it done properly?
> >>
> >> - Grant
> >>
> >>
> >> >> > And if I pull, none of my backed-up systems are secure because
> >> >> > anyone
> >> >> > who breaks into the backup server has root read privileges on
> >> >> > every
> >> >> > backed-up system and will thereby "gain full root privileges
> >> >> > quickly."
> >> >> 
> >> >>  IMO that depends on whether you also backup the
> >> >>  authentication-related
> >> >>  files or not. Exclude them from backup, ensure different root
> >> >>  passwords
> >> >>  for all boxes, and now you can limit the infiltration.
> >> >> >>>
> >> >> >>> If you're pulling to the backup server, that backup server has to
> >> >> >>> be
> >> >> >>> able to log in to and read all files on the other servers.
> >> >> >>> Including
> >> >> >>> e.g. your swap partition and device files.
> >> >> >>
> >> >> >> What if I have each system save a copy of everything to be backed
> up
> >> >> >> from its own filesystem in a separate directory and change the
> >> >> >> ownership of everything in that directory so it can be read by an
> >> >> >> unprivileged backup user?  Then I could have the backup server
> pull
> >> >> >> that copy from each system without giving it root access to each
> >> >> >> system.  Can I somehow have the correct ownerships for the backup
> >> >> >> saved in a separate file for use during a restore?
> >> >> >>
> >> >> >> - Grant
> >> >> >>
> >> >> >
> >> >> > You could just as well use an NFS share with no_root_squash. It is
> >> >> > really more a question of finding the right combination of tools to
> >> >> > ensure proper separation of concern for server and client.
> >> >> >
> >> >> > In fact, I think we are intermixing three distinct problems:
> >> >> > 1. (Possible) limitations of rdiff-backup with regard to untrusted
> >> >> > backup servers or clients.
> >> >>
> >> >> The limitation is real unfortunately.  All backups created by
> >> >> rdiff-backup more than a second ago can be deleted something like
> >> >> this:
> >> >>
> >> >> rdiff-backup --remove-older-than 1s backup@12.34.56.78:
> :/path/to/backup
> >> >>
> >> >> > 2. The purely technical question which file transfer protocols
> >> >> > protect
> >> >> > against write access from backup server to backup client and backup
> >> >> > client to older backups on the server.
> >> >>
> >> >> rdiff-backup doesn't provide those sort of protections.  Do any file
> >> >> transfer protocols?
> >> >>
> >> >> > 3. The more or less organisational question what level of
> protection
> >> >> > backups need and how fast security breaks have to be detected.
> >> >>
> >> >> I think backups should be very well protected and security breaks
> >> >> should not have to be immediately detected.
> >> >>
> >> >> - Grant
> >> >>
> >> >>
> >> >> > I think push vs. pull is just a secondary concern with regard to
> the
> >> >> > second question and has practically no relevance to the third one.
>

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Grant]

> You identified a flaw in the system as you were using it. You're right,
> those are flaws. However, you can " fix" those flaws by applying some magic
> as a sysadmin. That's why several posts in the thread have mentioned
> versioning your backups in some fashion. I've mentioned lvm a couple times.

I thought versioning meant that you could roll back to a previous
version.  rdiff-backup provides that.

> I think someone else mentioned pulling the backup target's data to another
> locale, either via a pull from another server, or via something like a
> traditional incremental tape backup.

So the systems push to the backup server and a second backup server
pulls from the first backup server?  Should the second backup server
use rdiff-backup against the rdiff-backup repository on the first
backup server?  I think I've read that's not a good idea.

What does everybody else do?  I feel like the first person to ever
attempt secure automated backups.

- Grant


> You're getting the data off the original machines to a remote location,
> which is good. You identified a way the backed-up data could be tampered
> with, which is good. You just need to put in another (better) barrier to
> protect the data from being tampered with, or limit how much data is lost in
> such an event.
>
> ZZ
>
> On Nov 14, 2011 8:21 PM, "Grant"  wrote:
>>
>> > It's out of scope for file transfer protocols; it's a
>> > daemon/system-local
>> > problem.  Attach pre-event or post-event scripts serverside for any
>> > special
>> > munging or protections you'd like to apply. (Such as triggering an LVM
>> > snapshot, for example...)
>>
>> I must be going about this the wrong way.  Am I the only one using
>> automated backups?  If not, how is it done properly?
>>
>> - Grant
>>
>>
>> >> > And if I pull, none of my backed-up systems are secure because
>> >> > anyone
>> >> > who breaks into the backup server has root read privileges on
>> >> > every
>> >> > backed-up system and will thereby "gain full root privileges
>> >> > quickly."
>> >> 
>> >>  IMO that depends on whether you also backup the
>> >>  authentication-related
>> >>  files or not. Exclude them from backup, ensure different root
>> >>  passwords
>> >>  for all boxes, and now you can limit the infiltration.
>> >> >>>
>> >> >>> If you're pulling to the backup server, that backup server has to
>> >> >>> be
>> >> >>> able to log in to and read all files on the other servers.
>> >> >>> Including
>> >> >>> e.g. your swap partition and device files.
>> >> >>
>> >> >> What if I have each system save a copy of everything to be backed up
>> >> >> from its own filesystem in a separate directory and change the
>> >> >> ownership of everything in that directory so it can be read by an
>> >> >> unprivileged backup user?  Then I could have the backup server pull
>> >> >> that copy from each system without giving it root access to each
>> >> >> system.  Can I somehow have the correct ownerships for the backup
>> >> >> saved in a separate file for use during a restore?
>> >> >>
>> >> >> - Grant
>> >> >>
>> >> >
>> >> > You could just as well use an NFS share with no_root_squash. It is
>> >> > really more a question of finding the right combination of tools to
>> >> > ensure proper separation of concern for server and client.
>> >> >
>> >> > In fact, I think we are intermixing three distinct problems:
>> >> > 1. (Possible) limitations of rdiff-backup with regard to untrusted
>> >> > backup servers or clients.
>> >>
>> >> The limitation is real unfortunately.  All backups created by
>> >> rdiff-backup more than a second ago can be deleted something like
>> >> this:
>> >>
>> >> rdiff-backup --remove-older-than 1s backup@12.34.56.78::/path/to/backup
>> >>
>> >> > 2. The purely technical question which file transfer protocols
>> >> > protect
>> >> > against write access from backup server to backup client and backup
>> >> > client to older backups on the server.
>> >>
>> >> rdiff-backup doesn't provide those sort of protections.  Do any file
>> >> transfer protocols?
>> >>
>> >> > 3. The more or less organisational question what level of protection
>> >> > backups need and how fast security breaks have to be detected.
>> >>
>> >> I think backups should be very well protected and security breaks
>> >> should not have to be immediately detected.
>> >>
>> >> - Grant
>> >>
>> >>
>> >> > I think push vs. pull is just a secondary concern with regard to the
>> >> > second question and has practically no relevance to the third one.
>> >> >
>> >> > Regards,
>> >> > Florian Philipp

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Michael Mol]

You identified a flaw in the system as you were using it. You're right,
those are flaws. However, you can " fix" those flaws by applying some magic
as a sysadmin. That's why several posts in the thread have mentioned
versioning your backups in some fashion. I've mentioned lvm a couple times.
I think someone else mentioned pulling the backup target's data to another
locale, either via a pull from another server, or via something like a
traditional incremental tape backup.

You're getting the data off the original machines to a remote location,
which is good. You identified a way the backed-up data could be tampered
with, which is good. You just need to put in another (better) barrier to
protect the data from being tampered with, or limit how much data is lost
in such an event.

ZZ
On Nov 14, 2011 8:21 PM, "Grant"  wrote:

> > It's out of scope for file transfer protocols; it's a daemon/system-local
> > problem.  Attach pre-event or post-event scripts serverside for any
> special
> > munging or protections you'd like to apply. (Such as triggering an LVM
> > snapshot, for example...)
>
> I must be going about this the wrong way.  Am I the only one using
> automated backups?  If not, how is it done properly?
>
> - Grant
>
>
> >> > And if I pull, none of my backed-up systems are secure because
> >> > anyone
> >> > who breaks into the backup server has root read privileges on
> every
> >> > backed-up system and will thereby "gain full root privileges
> >> > quickly."
> >> 
> >>  IMO that depends on whether you also backup the
> >>  authentication-related
> >>  files or not. Exclude them from backup, ensure different root
> >>  passwords
> >>  for all boxes, and now you can limit the infiltration.
> >> >>>
> >> >>> If you're pulling to the backup server, that backup server has to be
> >> >>> able to log in to and read all files on the other servers. Including
> >> >>> e.g. your swap partition and device files.
> >> >>
> >> >> What if I have each system save a copy of everything to be backed up
> >> >> from its own filesystem in a separate directory and change the
> >> >> ownership of everything in that directory so it can be read by an
> >> >> unprivileged backup user?  Then I could have the backup server pull
> >> >> that copy from each system without giving it root access to each
> >> >> system.  Can I somehow have the correct ownerships for the backup
> >> >> saved in a separate file for use during a restore?
> >> >>
> >> >> - Grant
> >> >>
> >> >
> >> > You could just as well use an NFS share with no_root_squash. It is
> >> > really more a question of finding the right combination of tools to
> >> > ensure proper separation of concern for server and client.
> >> >
> >> > In fact, I think we are intermixing three distinct problems:
> >> > 1. (Possible) limitations of rdiff-backup with regard to untrusted
> >> > backup servers or clients.
> >>
> >> The limitation is real unfortunately.  All backups created by
> >> rdiff-backup more than a second ago can be deleted something like
> >> this:
> >>
> >> rdiff-backup --remove-older-than 1s backup@12.34.56.78::/path/to/backup
> >>
> >> > 2. The purely technical question which file transfer protocols protect
> >> > against write access from backup server to backup client and backup
> >> > client to older backups on the server.
> >>
> >> rdiff-backup doesn't provide those sort of protections.  Do any file
> >> transfer protocols?
> >>
> >> > 3. The more or less organisational question what level of protection
> >> > backups need and how fast security breaks have to be detected.
> >>
> >> I think backups should be very well protected and security breaks
> >> should not have to be immediately detected.
> >>
> >> - Grant
> >>
> >>
> >> > I think push vs. pull is just a secondary concern with regard to the
> >> > second question and has practically no relevance to the third one.
> >> >
> >> > Regards,
> >> > Florian Philipp
>
>

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Grant]

>>Then I could have the backup server pull
>> that copy from each system without giving it root access to each
>> system.  Can I somehow have the correct ownerships for the backup
>> saved in a separate file for use during a restore?
>>
>
> If you're intent on making a two-stage pull work; you can do it by
> creating a 'backups' user on your servers, and then using filesystem
> ACLs to grant backups+r to every file/directory you want to back up.
> That way, an attacker on the backup server can't decide to peruse the
> rest of your stuff.

I like that.  So use ACLs to grant access to the backups instead of
using ownership/permissions so that the ownership/permissions stay
intact.  I've never used ACLs.  Do they "override"
ownership/permissions?  In other words, if the ACL specifies backups+r
to a file owned by root that is chmod 700, "backups" can read it
anyway?

> The easiest method, though, is to just add a third stage. Either move
> the backups on the backup server to another directory after the backup
> job completes, or sync/burn/whatever them off-site. In this case the
> backup server can't access anything you don't give it, and the
> individual servers can't trash their backed-up data.

I don't see how that could work in an automated fashion.  Could you
give me an example?

- Grant

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Grant]

> It's out of scope for file transfer protocols; it's a daemon/system-local
> problem.  Attach pre-event or post-event scripts serverside for any special
> munging or protections you'd like to apply. (Such as triggering an LVM
> snapshot, for example...)

I must be going about this the wrong way.  Am I the only one using
automated backups?  If not, how is it done properly?

- Grant


>> > And if I pull, none of my backed-up systems are secure because
>> > anyone
>> > who breaks into the backup server has root read privileges on every
>> > backed-up system and will thereby "gain full root privileges
>> > quickly."
>> 
>>  IMO that depends on whether you also backup the
>>  authentication-related
>>  files or not. Exclude them from backup, ensure different root
>>  passwords
>>  for all boxes, and now you can limit the infiltration.
>> >>>
>> >>> If you're pulling to the backup server, that backup server has to be
>> >>> able to log in to and read all files on the other servers. Including
>> >>> e.g. your swap partition and device files.
>> >>
>> >> What if I have each system save a copy of everything to be backed up
>> >> from its own filesystem in a separate directory and change the
>> >> ownership of everything in that directory so it can be read by an
>> >> unprivileged backup user?  Then I could have the backup server pull
>> >> that copy from each system without giving it root access to each
>> >> system.  Can I somehow have the correct ownerships for the backup
>> >> saved in a separate file for use during a restore?
>> >>
>> >> - Grant
>> >>
>> >
>> > You could just as well use an NFS share with no_root_squash. It is
>> > really more a question of finding the right combination of tools to
>> > ensure proper separation of concern for server and client.
>> >
>> > In fact, I think we are intermixing three distinct problems:
>> > 1. (Possible) limitations of rdiff-backup with regard to untrusted
>> > backup servers or clients.
>>
>> The limitation is real unfortunately.  All backups created by
>> rdiff-backup more than a second ago can be deleted something like
>> this:
>>
>> rdiff-backup --remove-older-than 1s backup@12.34.56.78::/path/to/backup
>>
>> > 2. The purely technical question which file transfer protocols protect
>> > against write access from backup server to backup client and backup
>> > client to older backups on the server.
>>
>> rdiff-backup doesn't provide those sort of protections.  Do any file
>> transfer protocols?
>>
>> > 3. The more or less organisational question what level of protection
>> > backups need and how fast security breaks have to be detected.
>>
>> I think backups should be very well protected and security breaks
>> should not have to be immediately detected.
>>
>> - Grant
>>
>>
>> > I think push vs. pull is just a secondary concern with regard to the
>> > second question and has practically no relevance to the third one.
>> >
>> > Regards,
>> > Florian Philipp

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Michael Mol]

It's out of scope for file transfer protocols; it's a daemon/system-local
problem.  Attach pre-event or post-event scripts serverside for any special
munging or protections you'd like to apply. (Such as triggering an LVM
snapshot, for example...)

(sorry for the top post; in-line can be done in this client, but it's more
cumbersome than I have time for atm...)

ZZ
On Nov 14, 2011 7:45 PM, "Grant"  wrote:

> > And if I pull, none of my backed-up systems are secure because anyone
> > who breaks into the backup server has root read privileges on every
> > backed-up system and will thereby "gain full root privileges
> quickly."
> 
>  IMO that depends on whether you also backup the authentication-related
>  files or not. Exclude them from backup, ensure different root
> passwords
>  for all boxes, and now you can limit the infiltration.
> >>>
> >>> If you're pulling to the backup server, that backup server has to be
> >>> able to log in to and read all files on the other servers. Including
> >>> e.g. your swap partition and device files.
> >>
> >> What if I have each system save a copy of everything to be backed up
> >> from its own filesystem in a separate directory and change the
> >> ownership of everything in that directory so it can be read by an
> >> unprivileged backup user?  Then I could have the backup server pull
> >> that copy from each system without giving it root access to each
> >> system.  Can I somehow have the correct ownerships for the backup
> >> saved in a separate file for use during a restore?
> >>
> >> - Grant
> >>
> >
> > You could just as well use an NFS share with no_root_squash. It is
> > really more a question of finding the right combination of tools to
> > ensure proper separation of concern for server and client.
> >
> > In fact, I think we are intermixing three distinct problems:
> > 1. (Possible) limitations of rdiff-backup with regard to untrusted
> > backup servers or clients.
>
> The limitation is real unfortunately.  All backups created by
> rdiff-backup more than a second ago can be deleted something like
> this:
>
> rdiff-backup --remove-older-than 1s backup@12.34.56.78::/path/to/backup
>
> > 2. The purely technical question which file transfer protocols protect
> > against write access from backup server to backup client and backup
> > client to older backups on the server.
>
> rdiff-backup doesn't provide those sort of protections.  Do any file
> transfer protocols?
>
> > 3. The more or less organisational question what level of protection
> > backups need and how fast security breaks have to be detected.
>
> I think backups should be very well protected and security breaks
> should not have to be immediately detected.
>
> - Grant
>
>
> > I think push vs. pull is just a secondary concern with regard to the
> > second question and has practically no relevance to the third one.
> >
> > Regards,
> > Florian Philipp
>
>

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Grant]

> And if I pull, none of my backed-up systems are secure because anyone
> who breaks into the backup server has root read privileges on every
> backed-up system and will thereby "gain full root privileges quickly."

 IMO that depends on whether you also backup the authentication-related
 files or not. Exclude them from backup, ensure different root passwords
 for all boxes, and now you can limit the infiltration.
>>>
>>> If you're pulling to the backup server, that backup server has to be
>>> able to log in to and read all files on the other servers. Including
>>> e.g. your swap partition and device files.
>>
>> What if I have each system save a copy of everything to be backed up
>> from its own filesystem in a separate directory and change the
>> ownership of everything in that directory so it can be read by an
>> unprivileged backup user?
>
> You've just reinvented the push backup =)
>
> If separate-directory is on the same server, an attacker can log in and
> overwrite all of your files with zeros. Those zeros will be pulled to
> the backup server, destroying your backups.

That's not the case at all.  The zeros would be pulled to the backup
server via rdiff-backup and saved as a new version in the repository.
The backups would be safe.

- Grant


> If separate-directory is on the backup server...

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Grant]

> And if I pull, none of my backed-up systems are secure because anyone
> who breaks into the backup server has root read privileges on every
> backed-up system and will thereby "gain full root privileges quickly."

 IMO that depends on whether you also backup the authentication-related
 files or not. Exclude them from backup, ensure different root passwords
 for all boxes, and now you can limit the infiltration.
>>>
>>> If you're pulling to the backup server, that backup server has to be
>>> able to log in to and read all files on the other servers. Including
>>> e.g. your swap partition and device files.
>>
>> What if I have each system save a copy of everything to be backed up
>> from its own filesystem in a separate directory and change the
>> ownership of everything in that directory so it can be read by an
>> unprivileged backup user?  Then I could have the backup server pull
>> that copy from each system without giving it root access to each
>> system.  Can I somehow have the correct ownerships for the backup
>> saved in a separate file for use during a restore?
>>
>> - Grant
>>
>
> You could just as well use an NFS share with no_root_squash. It is
> really more a question of finding the right combination of tools to
> ensure proper separation of concern for server and client.
>
> In fact, I think we are intermixing three distinct problems:
> 1. (Possible) limitations of rdiff-backup with regard to untrusted
> backup servers or clients.

The limitation is real unfortunately.  All backups created by
rdiff-backup more than a second ago can be deleted something like
this:

rdiff-backup --remove-older-than 1s backup@12.34.56.78::/path/to/backup

> 2. The purely technical question which file transfer protocols protect
> against write access from backup server to backup client and backup
> client to older backups on the server.

rdiff-backup doesn't provide those sort of protections.  Do any file
transfer protocols?

> 3. The more or less organisational question what level of protection
> backups need and how fast security breaks have to be detected.

I think backups should be very well protected and security breaks
should not have to be immediately detected.

- Grant


> I think push vs. pull is just a secondary concern with regard to the
> second question and has practically no relevance to the third one.
>
> Regards,
> Florian Philipp

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Michael Orlitzky]

On 11/13/11 13:03, Grant wrote:
> 
>Then I could have the backup server pull
> that copy from each system without giving it root access to each
> system.  Can I somehow have the correct ownerships for the backup
> saved in a separate file for use during a restore?
> 

If you're intent on making a two-stage pull work; you can do it by
creating a 'backups' user on your servers, and then using filesystem
ACLs to grant backups+r to every file/directory you want to back up.
That way, an attacker on the backup server can't decide to peruse the
rest of your stuff.

The easiest method, though, is to just add a third stage. Either move
the backups on the backup server to another directory after the backup
job completes, or sync/burn/whatever them off-site. In this case the
backup server can't access anything you don't give it, and the
individual servers can't trash their backed-up data.

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Michael Orlitzky]

On 11/13/11 13:03, Grant wrote:
 And if I pull, none of my backed-up systems are secure because anyone
 who breaks into the backup server has root read privileges on every
 backed-up system and will thereby "gain full root privileges quickly."
>>>
>>> IMO that depends on whether you also backup the authentication-related
>>> files or not. Exclude them from backup, ensure different root passwords
>>> for all boxes, and now you can limit the infiltration.
>>
>> If you're pulling to the backup server, that backup server has to be
>> able to log in to and read all files on the other servers. Including
>> e.g. your swap partition and device files.
> 
> What if I have each system save a copy of everything to be backed up
> from its own filesystem in a separate directory and change the
> ownership of everything in that directory so it can be read by an
> unprivileged backup user?

You've just reinvented the push backup =)

If separate-directory is on the same server, an attacker can log in and
overwrite all of your files with zeros. Those zeros will be pulled to
the backup server, destroying your backups.

If separate-directory is on the backup server...

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Florian Philipp]

Am 13.11.2011 19:03, schrieb Grant:
 And if I pull, none of my backed-up systems are secure because anyone
 who breaks into the backup server has root read privileges on every
 backed-up system and will thereby "gain full root privileges quickly."
>>>
>>> IMO that depends on whether you also backup the authentication-related
>>> files or not. Exclude them from backup, ensure different root passwords
>>> for all boxes, and now you can limit the infiltration.
>>
>> If you're pulling to the backup server, that backup server has to be
>> able to log in to and read all files on the other servers. Including
>> e.g. your swap partition and device files.
> 
> What if I have each system save a copy of everything to be backed up
> from its own filesystem in a separate directory and change the
> ownership of everything in that directory so it can be read by an
> unprivileged backup user?  Then I could have the backup server pull
> that copy from each system without giving it root access to each
> system.  Can I somehow have the correct ownerships for the backup
> saved in a separate file for use during a restore?
> 
> - Grant
> 

You could just as well use an NFS share with no_root_squash. It is
really more a question of finding the right combination of tools to
ensure proper separation of concern for server and client.

In fact, I think we are intermixing three distinct problems:
1. (Possible) limitations of rdiff-backup with regard to untrusted
backup servers or clients.
2. The purely technical question which file transfer protocols protect
against write access from backup server to backup client and backup
client to older backups on the server.
3. The more or less organisational question what level of protection
backups need and how fast security breaks have to be detected.

I think push vs. pull is just a secondary concern with regard to the
second question and has practically no relevance to the third one.

Regards,
Florian Philipp



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Grant]

>>> And if I pull, none of my backed-up systems are secure because anyone
>>> who breaks into the backup server has root read privileges on every
>>> backed-up system and will thereby "gain full root privileges quickly."
>>
>> IMO that depends on whether you also backup the authentication-related
>> files or not. Exclude them from backup, ensure different root passwords
>> for all boxes, and now you can limit the infiltration.
>
> If you're pulling to the backup server, that backup server has to be
> able to log in to and read all files on the other servers. Including
> e.g. your swap partition and device files.

What if I have each system save a copy of everything to be backed up
from its own filesystem in a separate directory and change the
ownership of everything in that directory so it can be read by an
unprivileged backup user?  Then I could have the backup server pull
that copy from each system without giving it root access to each
system.  Can I somehow have the correct ownerships for the backup
saved in a separate file for use during a restore?

- Grant

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Pandu Poluan]

On Nov 12, 2011 11:23 AM, "Michael Orlitzky"  wrote:
>
> On 11/11/2011 10:20 PM, Pandu Poluan wrote:
> >
> >> And if I pull, none of my backed-up systems are secure because anyone
> >> who breaks into the backup server has root read privileges on every
> >> backed-up system and will thereby "gain full root privileges quickly."
> >
> > IMO that depends on whether you also backup the authentication-related
> > files or not. Exclude them from backup, ensure different root passwords
> > for all boxes, and now you can limit the infiltration.
>
> If you're pulling to the backup server, that backup server has to be
> able to log in to and read all files on the other servers. Including
> e.g. your swap partition and device files.
>

Again, that's a matter of implementation.

If the server doesn't access the client's filesystem directly but via an
agent (Bacula does this, for instance), the server's access will be limited
to what the agent provides.

Rgds,

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Michael Orlitzky]

On 11/11/2011 10:20 PM, Pandu Poluan wrote:
> 
>> And if I pull, none of my backed-up systems are secure because anyone
>> who breaks into the backup server has root read privileges on every
>> backed-up system and will thereby "gain full root privileges quickly."
> 
> IMO that depends on whether you also backup the authentication-related
> files or not. Exclude them from backup, ensure different root passwords
> for all boxes, and now you can limit the infiltration.

If you're pulling to the backup server, that backup server has to be
able to log in to and read all files on the other servers. Including
e.g. your swap partition and device files.

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Michael Orlitzky]

On 11/11/2011 09:22 PM, Grant wrote:
> 
> So if I push, I don't really have backups because anyone who breaks
> into the backed-up system can delete all of its backups like this:
> 
> rdiff-backup --remove-older-than 1s backup@12.34.56.78::/path/to/backup
> 
> And if I pull, none of my backed-up systems are secure because anyone
> who breaks into the backup server has root read privileges on every
> backed-up system and will thereby "gain full root privileges quickly."
> 

It's a false dichotomy[1], but sums up the trade-off between those two
options well enough.

The last "hacker" who tried to delete everything on my system was a
5.25in floppy. So, I'm biased towards the other case.


[1] Third option: choose push or pull, and ALSO make off-site read-only
backups of the backup server every once in a while.

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Pandu Poluan]

On Nov 12, 2011 2:17 AM, "Florian Philipp"  wrote:
> >
> > Just an illustration: My employer will soon do a PoC/Live Demo of this
> > product:
> >
> > http://www.atempo.com/products/liveBackup/features.asp
> >
> > Only an 'agent' lives inside the employee's workstation. It pushes all
> > writes to certain folders to the server, and able to request 'reverts'
> > to their local copy, but the server's archives are immutable.
> >
> > Unfortunately, said product only supports Windows and Macs. I'm still on
> > the lookout for something similar for Linux.
> >
> > (For pure text files, a git/mercurial server would be enough, though.)
> >
> > Rgds,
> >
>
> Isn't Bacula something like this?
>
http://www.bacula.org/en/dev-manual/main/main/What_is_Bacula.html#SECTION0022
>
> Hint: "File server" actually is the client that is backed up.
>

Thanks! I knew someone has created something similar for Linux, but the
name escaped my mind :-)

Rgds,

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Pandu Poluan]

On Nov 12, 2011 9:29 AM, "Grant"  wrote:
>
> >> The problem with my current push-style layout is that if one of the 3
> >> machines is compromised, the attacker can delete or alter the backup
> >> of the compromised machine on the backup server.  I can rsync the
> >> backups from the backup server to another machine, but if the backups
> >> are deleted or altered on the backup server, the rsync'ed copy on the
> >> next machine will also be deleted or altered.
> >>
> >> If I run a pull-style layout and the backup server is compromised, the
> >> attacker would have root read access to each of the 3 machines, but
> >> the attacker would already have access to backups from each of the 3
> >> machines stored on the backup server itself so that's not really an
> >> issue.  I would also have the added inconvenience of using openvpn or
> >> ssh -R for my laptop so the backup server can pull from it through any
> >> router.
> >
> > If an attacker can read the entire filesystem, he'll gain full root
> > privileges quickly.
>
> So if I push, I don't really have backups because anyone who breaks
> into the backed-up system can delete all of its backups like this:
>
> rdiff-backup --remove-older-than 1s backup@12.34.56.78::/path/to/backup
>

Write a daemon that immediately create hardlinks of the backed-up files in
a separate folder. Thus, even if rdiff decides to unlink everything, your
data are safe thanks to the nature of hardlinks. Optionally, have the same
daemon tarball the files (via the hardlinks) if you deem the revision
'permanent'.

> And if I pull, none of my backed-up systems are secure because anyone
> who breaks into the backup server has root read privileges on every
> backed-up system and will thereby "gain full root privileges quickly."

IMO that depends on whether you also backup the authentication-related
files or not. Exclude them from backup, ensure different root passwords for
all boxes, and now you can limit the infiltration.

Rgds,

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Grant]

>> The problem with my current push-style layout is that if one of the 3
>> machines is compromised, the attacker can delete or alter the backup
>> of the compromised machine on the backup server.  I can rsync the
>> backups from the backup server to another machine, but if the backups
>> are deleted or altered on the backup server, the rsync'ed copy on the
>> next machine will also be deleted or altered.
>>
>> If I run a pull-style layout and the backup server is compromised, the
>> attacker would have root read access to each of the 3 machines, but
>> the attacker would already have access to backups from each of the 3
>> machines stored on the backup server itself so that's not really an
>> issue.  I would also have the added inconvenience of using openvpn or
>> ssh -R for my laptop so the backup server can pull from it through any
>> router.
>
> If an attacker can read the entire filesystem, he'll gain full root
> privileges quickly.

So if I push, I don't really have backups because anyone who breaks
into the backed-up system can delete all of its backups like this:

rdiff-backup --remove-older-than 1s backup@12.34.56.78::/path/to/backup

And if I pull, none of my backed-up systems are secure because anyone
who breaks into the backup server has root read privileges on every
backed-up system and will thereby "gain full root privileges quickly."

- Grant

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Michael Orlitzky]

On 11/11/2011 12:55 PM, Grant wrote:
> 
> The problem with my current push-style layout is that if one of the 3
> machines is compromised, the attacker can delete or alter the backup
> of the compromised machine on the backup server.  I can rsync the
> backups from the backup server to another machine, but if the backups
> are deleted or altered on the backup server, the rsync'ed copy on the
> next machine will also be deleted or altered.
> 
> If I run a pull-style layout and the backup server is compromised, the
> attacker would have root read access to each of the 3 machines, but
> the attacker would already have access to backups from each of the 3
> machines stored on the backup server itself so that's not really an
> issue.  I would also have the added inconvenience of using openvpn or
> ssh -R for my laptop so the backup server can pull from it through any
> router.

If an attacker can read the entire filesystem, he'll gain full root
privileges quickly.

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Florian Philipp]

Am 11.11.2011 19:56, schrieb Pandu Poluan:
> 
> On Nov 12, 2011 1:39 AM, "Grant"  > wrote:
>>
>> >> A little while ago I set up an automated backup system to back up the
>> >> data from 3 machines to a backup server.  I decided to use a
>> >> push-style layout where the 3 machines push their data to the backup
>> >> server.  Public SSH keys for the 3 machines are stored on the backup
>> >> server and restricted to the rdiff-backup command.  Each of the 3
>> >> machines pushes their data to the backup server as a different user
>> >> and the top directory of each backup is chmod 700 to prevent any of
>> >> the 3 machines from reading or writing a backup from another machine.
>> >>
>> >> I've run into a problem with this layout that I can't seem to solve,
>> >> and I'm wondering if I should switch to a pull-style layout where the
>> >> backup server pulls data from each of the 3 machines.
>> >>
>> >> The problem with my current push-style layout is that if one of the 3
>> >> machines is compromised, the attacker can delete or alter the backup
>> >> of the compromised machine on the backup server.  I can rsync the
>> >> backups from the backup server to another machine, but if the backups
>> >> are deleted or altered on the backup server, the rsync'ed copy on the
>> >> next machine will also be deleted or altered.
>> >>
>> >> If I run a pull-style layout and the backup server is compromised, the
>> >> attacker would have root read access to each of the 3 machines, but
>> >> the attacker would already have access to backups from each of the 3
>> >> machines stored on the backup server itself so that's not really an
>> >> issue.  I would also have the added inconvenience of using openvpn or
>> >> ssh -R for my laptop so the backup server can pull from it through any
>> >> router.
>> >>
>> >> What do you think guys?  Are push-style backups flawed and
> unacceptable?
>> >>
>> >
>> > No, it's not flawed, as long as the implementation is right:
> versioning and
>> > deduplication.
>> >
>> > With versioning, an attacker (or infiltrator, in this matter) might
> try to
>> > taint the backup, but all she can do is just push a new version to the
>> > server. You can recover your data by reverting to a prior version.
>>
>> Is that true?  Wouldn't the infiltrator be able to craft some sort of
>> rdiff-backup command that deletes the entire backup?  I can't come up
>> with such a command myself, but I thought I was essentially giving
>> full read/write access of a system's backup to an infiltrator by
>> putting that system's public key on the backup server.  I do restrict
>> the key like command="rdiff-backup --server" but I didn't expect that
>> to completely prevent the backup from being wiped out.  Does it?
>>
>> - Grant
>>
>>
>> > The deduplication part is only to save storage space. It's less
> necessary if
>> > you have a robust versioning system that can categorize each push as
> either
>> > canonical/perpetual/permanent or ephemeral/temporary. The system can
> just
>> > discard old ephemeral pushes when storage becomes critical.
>>
> 
> Just an illustration: My employer will soon do a PoC/Live Demo of this
> product:
> 
> http://www.atempo.com/products/liveBackup/features.asp
> 
> Only an 'agent' lives inside the employee's workstation. It pushes all
> writes to certain folders to the server, and able to request 'reverts'
> to their local copy, but the server's archives are immutable.
> 
> Unfortunately, said product only supports Windows and Macs. I'm still on
> the lookout for something similar for Linux.
> 
> (For pure text files, a git/mercurial server would be enough, though.)
> 
> Rgds,
> 

Isn't Bacula something like this?
http://www.bacula.org/en/dev-manual/main/main/What_is_Bacula.html#SECTION0022

Hint: "File server" actually is the client that is backed up.



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Pandu Poluan]

On Nov 12, 2011 1:39 AM, "Grant"  wrote:
>
> >> A little while ago I set up an automated backup system to back up the
> >> data from 3 machines to a backup server.  I decided to use a
> >> push-style layout where the 3 machines push their data to the backup
> >> server.  Public SSH keys for the 3 machines are stored on the backup
> >> server and restricted to the rdiff-backup command.  Each of the 3
> >> machines pushes their data to the backup server as a different user
> >> and the top directory of each backup is chmod 700 to prevent any of
> >> the 3 machines from reading or writing a backup from another machine.
> >>
> >> I've run into a problem with this layout that I can't seem to solve,
> >> and I'm wondering if I should switch to a pull-style layout where the
> >> backup server pulls data from each of the 3 machines.
> >>
> >> The problem with my current push-style layout is that if one of the 3
> >> machines is compromised, the attacker can delete or alter the backup
> >> of the compromised machine on the backup server.  I can rsync the
> >> backups from the backup server to another machine, but if the backups
> >> are deleted or altered on the backup server, the rsync'ed copy on the
> >> next machine will also be deleted or altered.
> >>
> >> If I run a pull-style layout and the backup server is compromised, the
> >> attacker would have root read access to each of the 3 machines, but
> >> the attacker would already have access to backups from each of the 3
> >> machines stored on the backup server itself so that's not really an
> >> issue.  I would also have the added inconvenience of using openvpn or
> >> ssh -R for my laptop so the backup server can pull from it through any
> >> router.
> >>
> >> What do you think guys?  Are push-style backups flawed and
unacceptable?
> >>
> >
> > No, it's not flawed, as long as the implementation is right: versioning
and
> > deduplication.
> >
> > With versioning, an attacker (or infiltrator, in this matter) might try
to
> > taint the backup, but all she can do is just push a new version to the
> > server. You can recover your data by reverting to a prior version.
>
> Is that true?  Wouldn't the infiltrator be able to craft some sort of
> rdiff-backup command that deletes the entire backup?  I can't come up
> with such a command myself, but I thought I was essentially giving
> full read/write access of a system's backup to an infiltrator by
> putting that system's public key on the backup server.  I do restrict
> the key like command="rdiff-backup --server" but I didn't expect that
> to completely prevent the backup from being wiped out.  Does it?
>
> - Grant
>
>
> > The deduplication part is only to save storage space. It's less
necessary if
> > you have a robust versioning system that can categorize each push as
either
> > canonical/perpetual/permanent or ephemeral/temporary. The system can
just
> > discard old ephemeral pushes when storage becomes critical.
>

Just an illustration: My employer will soon do a PoC/Live Demo of this
product:

http://www.atempo.com/products/liveBackup/features.asp

Only an 'agent' lives inside the employee's workstation. It pushes all
writes to certain folders to the server, and able to request 'reverts' to
their local copy, but the server's archives are immutable.

Unfortunately, said product only supports Windows and Macs. I'm still on
the lookout for something similar for Linux.

(For pure text files, a git/mercurial server would be enough, though.)

Rgds,

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Grant]

>> A little while ago I set up an automated backup system to back up the
>> data from 3 machines to a backup server.  I decided to use a
>> push-style layout where the 3 machines push their data to the backup
>> server.  Public SSH keys for the 3 machines are stored on the backup
>> server and restricted to the rdiff-backup command.  Each of the 3
>> machines pushes their data to the backup server as a different user
>> and the top directory of each backup is chmod 700 to prevent any of
>> the 3 machines from reading or writing a backup from another machine.
>>
>> I've run into a problem with this layout that I can't seem to solve,
>> and I'm wondering if I should switch to a pull-style layout where the
>> backup server pulls data from each of the 3 machines.
>>
>> The problem with my current push-style layout is that if one of the 3
>> machines is compromised, the attacker can delete or alter the backup
>> of the compromised machine on the backup server.  I can rsync the
>> backups from the backup server to another machine, but if the backups
>> are deleted or altered on the backup server, the rsync'ed copy on the
>> next machine will also be deleted or altered.
>>
>> If I run a pull-style layout and the backup server is compromised, the
>> attacker would have root read access to each of the 3 machines, but
>> the attacker would already have access to backups from each of the 3
>> machines stored on the backup server itself so that's not really an
>> issue.  I would also have the added inconvenience of using openvpn or
>> ssh -R for my laptop so the backup server can pull from it through any
>> router.
>>
>> What do you think guys?  Are push-style backups flawed and unacceptable?
>>
>
> No, it's not flawed, as long as the implementation is right: versioning and
> deduplication.
>
> With versioning, an attacker (or infiltrator, in this matter) might try to
> taint the backup, but all she can do is just push a new version to the
> server. You can recover your data by reverting to a prior version.

Is that true?  Wouldn't the infiltrator be able to craft some sort of
rdiff-backup command that deletes the entire backup?  I can't come up
with such a command myself, but I thought I was essentially giving
full read/write access of a system's backup to an infiltrator by
putting that system's public key on the backup server.  I do restrict
the key like command="rdiff-backup --server" but I didn't expect that
to completely prevent the backup from being wiped out.  Does it?

- Grant


> The deduplication part is only to save storage space. It's less necessary if
> you have a robust versioning system that can categorize each push as either
> canonical/perpetual/permanent or ephemeral/temporary. The system can just
> discard old ephemeral pushes when storage becomes critical.

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Michael Mol]

On Fri, Nov 11, 2011 at 1:27 PM, Grant  wrote:
>> [snip]
>>
>>> The problem with my current push-style layout is that if one of the 3
>>> machines is compromised, the attacker can delete or alter the backup
>>> of the compromised machine on the backup server.  I can rsync the
>>> backups from the backup server to another machine, but if the backups
>>> are deleted or altered on the backup server, the rsync'ed copy on the
>>> next machine will also be deleted or altered.
>>
>> As a final stage in your backup, could you trigger a 'pull'-style
>> backup copying the data image to a more secure area? How about setting
>
> Even if I pull a copy of the backup to a separate machine from the
> backup server, it will pull an altered copy if an attacker compromises
> one of the systems being backed up and alters that system's backup on
> the backup server.  Am I missing something?

If you're not applying any kind of versioning, it doesn't matter if
you're pushing or pulling; your backup will eventually be overwritten
by a backup of a hacked system unless you catch and respond as soon as
the original invasion happens. So it sounds like the scenario you fear
isn't tied to the mechanism you're reconsidering.

-- 
:wq

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Grant]

> [snip]
>
>> The problem with my current push-style layout is that if one of the 3
>> machines is compromised, the attacker can delete or alter the backup
>> of the compromised machine on the backup server.  I can rsync the
>> backups from the backup server to another machine, but if the backups
>> are deleted or altered on the backup server, the rsync'ed copy on the
>> next machine will also be deleted or altered.
>
> As a final stage in your backup, could you trigger a 'pull'-style
> backup copying the data image to a more secure area? How about setting

Even if I pull a copy of the backup to a separate machine from the
backup server, it will pull an altered copy if an attacker compromises
one of the systems being backed up and alters that system's backup on
the backup server.  Am I missing something?

- Grant


> your backup target on top of lvm, and snapshotting? Some mechanism
> could be employed so that the snapshot command is run by a more
> restricted user, and done so after, e.g. a certain amount of idle time
> in the backup target directory
>
>>
>> If I run a pull-style layout and the backup server is compromised, the
>> attacker would have root read access to each of the 3 machines, but
>> the attacker would already have access to backups from each of the 3
>> machines stored on the backup server itself so that's not really an
>> issue.  I would also have the added inconvenience of using openvpn or
>> ssh -R for my laptop so the backup server can pull from it through any
>> router.
>
> Check out freenet6. I use it so that my laptop has a static, global IP
> address whether it's on my home network or not. It's quite nice. IPv6
> in various applications also solves my other direct-access needs.
>
>>
>> What do you think guys?  Are push-style backups flawed and unacceptable?
>
> I imagine you might still want to 'pull' from your backup server; if
> someone gets a key that allows them to manipulate the behavior of a
> local process that shouldn't normally be manipulated, your
> vulnerability surface goes up.
>
> --
> :wq

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Pandu Poluan]

On Nov 12, 2011 12:58 AM, "Grant"  wrote:
>
> A little while ago I set up an automated backup system to back up the
> data from 3 machines to a backup server.  I decided to use a
> push-style layout where the 3 machines push their data to the backup
> server.  Public SSH keys for the 3 machines are stored on the backup
> server and restricted to the rdiff-backup command.  Each of the 3
> machines pushes their data to the backup server as a different user
> and the top directory of each backup is chmod 700 to prevent any of
> the 3 machines from reading or writing a backup from another machine.
>
> I've run into a problem with this layout that I can't seem to solve,
> and I'm wondering if I should switch to a pull-style layout where the
> backup server pulls data from each of the 3 machines.
>
> The problem with my current push-style layout is that if one of the 3
> machines is compromised, the attacker can delete or alter the backup
> of the compromised machine on the backup server.  I can rsync the
> backups from the backup server to another machine, but if the backups
> are deleted or altered on the backup server, the rsync'ed copy on the
> next machine will also be deleted or altered.
>
> If I run a pull-style layout and the backup server is compromised, the
> attacker would have root read access to each of the 3 machines, but
> the attacker would already have access to backups from each of the 3
> machines stored on the backup server itself so that's not really an
> issue.  I would also have the added inconvenience of using openvpn or
> ssh -R for my laptop so the backup server can pull from it through any
> router.
>
> What do you think guys?  Are push-style backups flawed and unacceptable?
>

No, it's not flawed, as long as the implementation is right: versioning and
deduplication.

With versioning, an attacker (or infiltrator, in this matter) might try to
taint the backup, but all she can do is just push a new version to the
server. You can recover your data by reverting to a prior version.

The deduplication part is only to save storage space. It's less necessary
if you have a robust versioning system that can categorize each push as
either canonical/perpetual/permanent or ephemeral/temporary. The system can
just discard old ephemeral pushes when storage becomes critical.

Rgds,

Re: [gentoo-user] {OT} Are "push" backups flawed?

[Michael Mol]

On Fri, Nov 11, 2011 at 12:55 PM, Grant  wrote:

[snip]

> The problem with my current push-style layout is that if one of the 3
> machines is compromised, the attacker can delete or alter the backup
> of the compromised machine on the backup server.  I can rsync the
> backups from the backup server to another machine, but if the backups
> are deleted or altered on the backup server, the rsync'ed copy on the
> next machine will also be deleted or altered.

As a final stage in your backup, could you trigger a 'pull'-style
backup copying the data image to a more secure area? How about setting
your backup target on top of lvm, and snapshotting? Some mechanism
could be employed so that the snapshot command is run by a more
restricted user, and done so after, e.g. a certain amount of idle time
in the backup target directory

>
> If I run a pull-style layout and the backup server is compromised, the
> attacker would have root read access to each of the 3 machines, but
> the attacker would already have access to backups from each of the 3
> machines stored on the backup server itself so that's not really an
> issue.  I would also have the added inconvenience of using openvpn or
> ssh -R for my laptop so the backup server can pull from it through any
> router.

Check out freenet6. I use it so that my laptop has a static, global IP
address whether it's on my home network or not. It's quite nice. IPv6
in various applications also solves my other direct-access needs.

>
> What do you think guys?  Are push-style backups flawed and unacceptable?

I imagine you might still want to 'pull' from your backup server; if
someone gets a key that allows them to manipulate the behavior of a
local process that shouldn't normally be manipulated, your
vulnerability surface goes up.

-- 
:wq

[gentoo-user] {OT} Are "push" backups flawed?

[Grant]

A little while ago I set up an automated backup system to back up the
data from 3 machines to a backup server.  I decided to use a
push-style layout where the 3 machines push their data to the backup
server.  Public SSH keys for the 3 machines are stored on the backup
server and restricted to the rdiff-backup command.  Each of the 3
machines pushes their data to the backup server as a different user
and the top directory of each backup is chmod 700 to prevent any of
the 3 machines from reading or writing a backup from another machine.

I've run into a problem with this layout that I can't seem to solve,
and I'm wondering if I should switch to a pull-style layout where the
backup server pulls data from each of the 3 machines.

The problem with my current push-style layout is that if one of the 3
machines is compromised, the attacker can delete or alter the backup
of the compromised machine on the backup server.  I can rsync the
backups from the backup server to another machine, but if the backups
are deleted or altered on the backup server, the rsync'ed copy on the
next machine will also be deleted or altered.

If I run a pull-style layout and the backup server is compromised, the
attacker would have root read access to each of the 3 machines, but
the attacker would already have access to backups from each of the 3
machines stored on the backup server itself so that's not really an
issue.  I would also have the added inconvenience of using openvpn or
ssh -R for my laptop so the backup server can pull from it through any
router.

What do you think guys?  Are push-style backups flawed and unacceptable?

- Grant