Simo Sorce wrote:
On Tue, 2013-02-19 at 22:43 -0500, Rob Crittenden wrote:
I've looked into some basic backup and restore procedures for IPA. My
findings are here: http://freeipa.org/page/V3/Backup_and_Restore
For the catastrofic failure scenario, should we mention how to put back
a full failed and restored machine online ? I am thinking the restored
server may be behind (even if only by a few entries) in the replication,
so the CSNs in the other replicas will not match.
I guess we should mention a full resync may be needed ? Or is there a
way to bring back CSNs on replicas ?
Good questions. It depends on how long the machine was down and how many
changes have happened. It is possible that one would want to do a full
re-init. I'll add that to the design.
In the 'Returning to a good state' case, can we consider some split
brain approach, were we sever replication and rebuild one server at a
Perhaps using a firewall, but then you run the risk of each of those
servers accepting changes during the rebuild and you could end up with a
lot of collisions, which sort of goes against the point of restoring to
a known good state.
The changelog is the key here. I'll have to ponder this one a bit, I'm a
bit conflicted on the right approach.
Maybe we can think of a way to 'mark' all server as 'bad' so that on
restore the replication agreements do not need to be changed but changes
from 'bad' servers will not be accepted ?
I guess this crosses also a request by someone to be able to 'pause'
replication, would use the same mechanism I guess.
AFAIK there is an option to pause replication now (at least in 1.3).
What you can't do is drop the changelog AFAIK. That is the real problem.
If you want to restore to a known state you need to drop all the
changelog entries since that time. I'll check with the 389-ds team to
see if that is possible. Since we know the time of the backup, we might
be able to drop newer entries than that.
Full system backup:
in the first part it is said the process is offline, but in the 'LDAP'
section you say ldapi is used, but that would mean DS is running ?
Also are we sure we can get all data we need via ldapi ? Are we going to
miss any "operational" attribute ?
The full backup is offline because it is just using tar. This is sort of
a brute-force backup, copying bits from A to B.
The data backup is online and creates a task in 389-ds to write the data
and changelog to a file. It should write everything completely. We don't
do an ldapsearch.
I chose to not back up in ldif because this would back up just the data
and not the changelog. The other advantage is that the db2bak format
includes the indexes and ldif restore would require a rebuild.
For restore are we sure we can reload data w/o alterations ? What about
plugins ? will we have to disable all plugins during a restore ?
Yes, it should be fine. I'm hoping that the version will help us with
this, to prevent someone from restoring an ancient backup on a new
system, for example (or the reverse).
For the open questions.
Size of backup:
I think we should make it easy to configure the system to use a custom
directory to dump the backups. This way admins can make sure it is on a
different partition/disk or even over NFS and that the backup will not
fill up the disk on which DS is running.
That's a good idea. I'll have to think about where we would configure
that. Perhaps as an optional argument to the backup command.
We should definitely allow to encrypt backup files, a gpg public key
would be sufficient.
Ok. I wasn't sure if there would be any corruption concerns.
For replica cases, maybe we can create a command that dumps the
changelog from a good replica and then allow us to reply all changes
that are missing from the backup to bring the server up to the last
This would happen when we went online anyway though, at least for the
entries currently in the changelog. I guess this would have the
advantage of doing it in bulk and not over a (potentially) slow link.
Freeipa-devel mailing list