Re: [Freeipa-users] disaster recovery
On Mon, 27 Jun 2016 08:59:14 -0400 Robert wrote: RS> On Mon, 27 Jun 2016 08:09:59 +0200 Martin wrote: RS> MB> On 26.06.2016 08:17, Robert Story wrote: RS> MB> > Hello, RS> MB> > RS> MB> > I was running a single ipa instance on Centos 7 for a small lab RS> MB> > (ipa-server-4.2.0-15.0.1.el7.centos.17.x86_64), and the disk was corrupted. RS> MB> > I have a (mostly) full backup (/var/log/ and /var/run/ excluded), which I RS> MB> > restored. ipa server didn't start, and wanted me to run RS> MB> > ipa-server-upgrade. This failed, and I see this in the log: RS> MB> > [...] RS> MB> Hello, upgrader refuses to upgrade because check which requires RS> MB> /var/lib/ipa failed. Upgrader thinks that IPA is not installed. RS> MB> RS> MB> So are you sure you have backup of /var/lib/ipa ? RS> RS> Yep, /var/lib/ipa is there: RS> RS> ls -lR RS> [...] RS> ./pki-ca/publish: RS> total 0 RS> lrwxrwxrwx. 1 pkiuser pkiuser 57 Jun 24 21:00 MasterCRL.bin -> /var/lib/ipa/pki-ca/publish/MasterCRL-20160624-21.der RS> RS> RS> Looking through the backups, I see that there are no MasterCRL files from RS> the 25th (the backup I restored), but a bunch from the 24th, so maybe I RS> need to try another restore with files from then... So restoring /var/lib/ipa didn't work, and restoring the whole VM is taking way to long. I have a new VM up with a new ipa-server install, and am wondering if there is a way to import the data from the old filesystem? Robert -- Senior Software Engineer @ Parsons pgpWdQ3DBFr2R.pgp Description: OpenPGP digital signature -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] disaster recovery
On Mon, 27 Jun 2016 08:09:59 +0200 Martin wrote:
MB> On 26.06.2016 08:17, Robert Story wrote:
MB> > Hello,
MB> >
MB> > I was running a single ipa instance on Centos 7 for a small lab
MB> > (ipa-server-4.2.0-15.0.1.el7.centos.17.x86_64), and the disk was
corrupted.
MB> > I have a (mostly) full backup (/var/log/ and /var/run/ excluded), which I
MB> > restored. ipa server didn't start, and wanted me to run
MB> > ipa-server-upgrade. This failed, and I see this in the log:
MB> >
MB> > 2016-06-25T23:16:37Z DEBUG Mounting ipaserver.rpcserver.jsonserver_kerb()
at '/json'
MB> > 2016-06-25T23:16:37Z DEBUG session_auth_duration: 0:20:00
MB> > 2016-06-25T23:16:37Z DEBUG Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
MB> > 2016-06-25T23:16:37Z DEBUG File
"/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in execute
MB> > return_value = self.run()
MB> >File
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
line 47, in run
MB> > server.upgrade_check(self.options)
MB> >File
"/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line
1573, in upgrade_check
MB> > sys.exit(1)
MB> >
MB> > 2016-06-25T23:16:37Z DEBUG The ipa-server-upgrade command failed,
exception: SystemExit: 1
MB> >
MB> >
MB> > I tried starting dirsrv@DOMAIN manually, and I get thisin the dirsrv log:
MB> >
MB> >
MB> > [26/Jun/2016:01:46:54 -0400] - 389-Directory/1.3.4.0 B2016.175.1716
starting up
MB> > [26/Jun/2016:01:46:54 -0400] - WARNING: changelog: entry cache size
2097152B is less than db size 143196160B; We recommend to increase the entry
cache size nsslapd-cachememsize.
MB> > [26/Jun/2016:01:46:54 -0400] - Detected Disorderly Shutdown last time
Directory Server was running, recovering database.
MB> > [26/Jun/2016:01:46:55 -0400] - libdb: BDB2506 file userRoot/id2entry.db
has LSN 4336/2969724, past end of log at 1/176
MB> > [26/Jun/2016:01:46:56 -0400] - libdb: BDB2507 Commonly caused by moving a
database from one database environment
MB> > [26/Jun/2016:01:46:56 -0400] - libdb: BDB2508 to another without clearing
the database LSNs, or by removing all of
MB> > [26/Jun/2016:01:46:56 -0400] - libdb: BDB2509 the log files from a
database environment
MB> > [26/Jun/2016:01:46:57 -0400] - dbp->open("userRoot/id2entry.db") failed:
Invalid argument (22)
MB> > [26/Jun/2016:01:46:57 -0400] - dblayer_instance_start fail: Invalid
argument (22)
MB> > [26/Jun/2016:01:46:57 -0400] - libdb: BDB2506 file ipaca/id2entry.db has
LSN 4336/2990140, past end of log at 1/288
MB> > [26/Jun/2016:01:46:57 -0400] - libdb: BDB2507 Commonly caused by moving a
database from one database environment
MB> > [26/Jun/2016:01:46:57 -0400] - libdb: BDB2508 to another without clearing
the database LSNs, or by removing all of
MB> > [26/Jun/2016:01:46:57 -0400] - libdb: BDB2509 the log files from a
database environment
MB> > [26/Jun/2016:01:46:57 -0400] - dbp->open("ipaca/id2entry.db") failed:
Invalid argument (22)
MB> > [26/Jun/2016:01:46:58 -0400] - dblayer_instance_start fail: Invalid
argument (22)
MB> > [26/Jun/2016:01:46:58 -0400] - libdb: BDB2506 file changelog/id2entry.db
has LSN 4336/2921967, past end of log at 1/288
MB> > [26/Jun/2016:01:46:58 -0400] - libdb: BDB2507 Commonly caused by moving a
database from one database environment
MB> > [26/Jun/2016:01:46:58 -0400] - libdb: BDB2508 to another without clearing
the database LSNs, or by removing all of
MB> > [26/Jun/2016:01:46:58 -0400] - libdb: BDB2509 the log files from a
database environment
MB> > [26/Jun/2016:01:46:58 -0400] - dbp->open("changelog/id2entry.db") failed:
Invalid argument (22)
MB> > [26/Jun/2016:01:46:58 -0400] - dblayer_instance_start fail: Invalid
argument (22)
MB> > [26/Jun/2016:01:46:58 -0400] - start: Failed to start databases, err=22
Invalid argument
MB> >
MB> >
MB> > So I'm trying to figure out if I can salvage this restored VM, or if I
need
MB> > to reinstall from scratch; and if I do reinstall, am I going to be able to
MB> > restore my old data somehow. I have a funny feeling that there are
MB> > important files in /var/log and/or /var/run and I'm up the creek without a
MB> > paddle.
MB> >
MB> > And yes, once I have a working system again I'm going to set up a replica
MB> > to help avoid this mess in the future.
MB> >
MB> > Robert
MB> >
MB> >
MB> >
MB>
MB> Hello, upgrader refuses to upgrade because check which requires
MB> /var/lib/ipa failed. Upgrader thinks that IPA is not installed.
MB>
MB> So are you sure you have backup of /var/lib/ipa ?
Yep, /var/lib/ipa is there:
ls -lR
.:
total 4
drwx--. 2 root root6 Jun 24 08:10 backup
drwxr-xr-x. 3 root root 20 Jun 24 08:10 pki-ca
drwx--. 2 root root 4096 Jun 24 08:10 sysrestore
drwx--. 2 root root 29 Jun 24 08:10 sysupgrade
./backup:
total 0
./pki-ca:
total 0
drwxrwxr-x. 2 root pkiuser 26 Jun 25 19:38 publish
./pki-ca/publish:
total 0
lrwxrwxrwx. 1 pkiuser pkiuser 57 Jun 24 21:00 MasterCRL.bin ->
Re: [Freeipa-users] disaster recovery
On 26.06.2016 08:17, Robert Story wrote:
Hello,
I was running a single ipa instance on Centos 7 for a small lab
(ipa-server-4.2.0-15.0.1.el7.centos.17.x86_64), and the disk was corrupted.
I have a (mostly) full backup (/var/log/ and /var/run/ excluded), which I
restored. ipa server didn't start, and wanted me to run
ipa-server-upgrade. This failed, and I see this in the log:
2016-06-25T23:16:37Z DEBUG Mounting ipaserver.rpcserver.jsonserver_kerb() at
'/json'
2016-06-25T23:16:37Z DEBUG session_auth_duration: 0:20:00
2016-06-25T23:16:37Z DEBUG Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
2016-06-25T23:16:37Z DEBUG File
"/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in execute
return_value = self.run()
File
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
line 47, in run
server.upgrade_check(self.options)
File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
line 1573, in upgrade_check
sys.exit(1)
2016-06-25T23:16:37Z DEBUG The ipa-server-upgrade command failed, exception:
SystemExit: 1
I tried starting dirsrv@DOMAIN manually, and I get thisin the dirsrv log:
[26/Jun/2016:01:46:54 -0400] - 389-Directory/1.3.4.0 B2016.175.1716 starting up
[26/Jun/2016:01:46:54 -0400] - WARNING: changelog: entry cache size 2097152B is
less than db size 143196160B; We recommend to increase the entry cache size
nsslapd-cachememsize.
[26/Jun/2016:01:46:54 -0400] - Detected Disorderly Shutdown last time Directory
Server was running, recovering database.
[26/Jun/2016:01:46:55 -0400] - libdb: BDB2506 file userRoot/id2entry.db has LSN
4336/2969724, past end of log at 1/176
[26/Jun/2016:01:46:56 -0400] - libdb: BDB2507 Commonly caused by moving a
database from one database environment
[26/Jun/2016:01:46:56 -0400] - libdb: BDB2508 to another without clearing the
database LSNs, or by removing all of
[26/Jun/2016:01:46:56 -0400] - libdb: BDB2509 the log files from a database
environment
[26/Jun/2016:01:46:57 -0400] - dbp->open("userRoot/id2entry.db") failed:
Invalid argument (22)
[26/Jun/2016:01:46:57 -0400] - dblayer_instance_start fail: Invalid argument
(22)
[26/Jun/2016:01:46:57 -0400] - libdb: BDB2506 file ipaca/id2entry.db has LSN
4336/2990140, past end of log at 1/288
[26/Jun/2016:01:46:57 -0400] - libdb: BDB2507 Commonly caused by moving a
database from one database environment
[26/Jun/2016:01:46:57 -0400] - libdb: BDB2508 to another without clearing the
database LSNs, or by removing all of
[26/Jun/2016:01:46:57 -0400] - libdb: BDB2509 the log files from a database
environment
[26/Jun/2016:01:46:57 -0400] - dbp->open("ipaca/id2entry.db") failed: Invalid
argument (22)
[26/Jun/2016:01:46:58 -0400] - dblayer_instance_start fail: Invalid argument
(22)
[26/Jun/2016:01:46:58 -0400] - libdb: BDB2506 file changelog/id2entry.db has
LSN 4336/2921967, past end of log at 1/288
[26/Jun/2016:01:46:58 -0400] - libdb: BDB2507 Commonly caused by moving a
database from one database environment
[26/Jun/2016:01:46:58 -0400] - libdb: BDB2508 to another without clearing the
database LSNs, or by removing all of
[26/Jun/2016:01:46:58 -0400] - libdb: BDB2509 the log files from a database
environment
[26/Jun/2016:01:46:58 -0400] - dbp->open("changelog/id2entry.db") failed:
Invalid argument (22)
[26/Jun/2016:01:46:58 -0400] - dblayer_instance_start fail: Invalid argument
(22)
[26/Jun/2016:01:46:58 -0400] - start: Failed to start databases, err=22 Invalid
argument
So I'm trying to figure out if I can salvage this restored VM, or if I need
to reinstall from scratch; and if I do reinstall, am I going to be able to
restore my old data somehow. I have a funny feeling that there are
important files in /var/log and/or /var/run and I'm up the creek without a
paddle.
And yes, once I have a working system again I'm going to set up a replica
to help avoid this mess in the future.
Robert
Hello, upgrader refuses to upgrade because check which requires
/var/lib/ipa failed. Upgrader thinks that IPA is not installed.
So are you sure you have backup of /var/lib/ipa ?
regards,
Martin
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] Disaster Recovery Best Practices?
On Apr 16, 2012, at 12:40 PM, Dmitri Pal wrote: 2) What is everyone else doing to prepare IPA for a DR? I've read that the best way to do it is to turn off the IPA services on a replica and then back that replica up. I also read that this will miss some important files that only exist on the master. That is the case when you use selfsigned cert but the preferred and default configuration is not with the self-signed certs. It was in the past but not any more. Currently when you install IPA and then replicas there is no difference between master and replicas (if you installed CA on the replica) so picking any one and recycling is possible. You won't loose anything. Can 389DS produce a full 'backup' in an LDIF of schema / objects while running? -Brian___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Disaster Recovery Best Practices?
On 04/20/2012 08:46 AM, Brian Cook wrote: On Apr 16, 2012, at 12:40 PM, Dmitri Pal wrote: 2) What is everyone else doing to prepare IPA for a DR? I've read that the best way to do it is to turn off the IPA services on a replica and then back that replica up. I also read that this will miss some important files that only exist on the master. That is the case when you use selfsigned cert but the preferred and default configuration is not with the self-signed certs. It was in the past but not any more. Currently when you install IPA and then replicas there is no difference between master and replicas (if you installed CA on the replica) so picking any one and recycling is possible. You won't loose anything. Can 389DS produce a full 'backup' in an LDIF of schema / objects while running? While running - yes Here is a document that describes 389 database management: http://docs.redhat.com/docs/en-US/Red_Hat_Directory_Server/9.0/html/Administration_Guide/Populating_Directory_Databases.html Schema files can just be copied/tarred from /etc/dirsrv/slapd-*/schema The real question is - how does this work with IPA? -Brian ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Disaster Recovery Best Practices?
On 04/20/2012 11:47 AM, Rich Megginson wrote: On 04/20/2012 08:46 AM, Brian Cook wrote: On Apr 16, 2012, at 12:40 PM, Dmitri Pal wrote: 2) What is everyone else doing to prepare IPA for a DR? I've read that the best way to do it is to turn off the IPA services on a replica and then back that replica up. I also read that this will miss some important files that only exist on the master. That is the case when you use selfsigned cert but the preferred and default configuration is not with the self-signed certs. It was in the past but not any more. Currently when you install IPA and then replicas there is no difference between master and replicas (if you installed CA on the replica) so picking any one and recycling is possible. You won't loose anything. Can 389DS produce a full 'backup' in an LDIF of schema / objects while running? While running - yes Here is a document that describes 389 database management: http://docs.redhat.com/docs/en-US/Red_Hat_Directory_Server/9.0/html/Administration_Guide/Populating_Directory_Databases.html Schema files can just be copied/tarred from /etc/dirsrv/slapd-*/schema The real question is - how does this work with IPA? The problem is that there are config files, certificates in the NSS database that also need to be backed up to be able to restore the system. It is easy to just stand up a new replica instead of the lost one than to collect data and then try to restore. -Brian ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users -- Thank you, Dmitri Pal Sr. Engineering Manager IPA project, Red Hat Inc. --- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Disaster Recovery Best Practices?
On 04/16/2012 03:13 PM, KodaK wrote: Hi, I have googled around a bit, but I still have a couple of questions: 1) is it possible to get getent shadow to return shadow entries from the ipa server? This is so we can do a DR test on some server or set of servers without also having to restore the IPA server first. I can do a getent passwd easily enough, and I could rebuild the shadow file for local users, so it's not critical, but it would be a nice to have in the case of a DR. Please use SSSD on the client. It will do all the caching for you. If the connection is lost to the central server the client will continue to operate and authenticate users that logged in previously at least once. There is no need to create shadow files on the client in this case. Shadow is a mistake of the past that should not be used when there are are other much more secure technologies available now. 2) What is everyone else doing to prepare IPA for a DR? I've read that the best way to do it is to turn off the IPA services on a replica and then back that replica up. I also read that this will miss some important files that only exist on the master. That is the case when you use selfsigned cert but the preferred and default configuration is not with the self-signed certs. It was in the past but not any more. Currently when you install IPA and then replicas there is no difference between master and replicas (if you installed CA on the replica) so picking any one and recycling is possible. You won't loose anything. I don't want to turn off the master server services for a DR due to failover lag. Would it be safe to take a backup of the master while hot, then restore a replica, and promote it to master using the hot backup of the master (just the specific CA files needed)? So turning off any server of your choice backing it up (taking a snapshot) and then re-starting it again is the simplest way of dealing with DR. But to do this make sure that the server that you plan to use for taking backup snapshots has a CA. Thanks, --Jason ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users -- Thank you, Dmitri Pal Sr. Engineering Manager IPA project, Red Hat Inc. --- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Disaster Recovery Best Practices?
On Mon, 2012-04-16 at 14:13 -0500, KodaK wrote: Hi, I have googled around a bit, but I still have a couple of questions: 1) is it possible to get getent shadow to return shadow entries from the ipa server? No, we do not have any shadow map in ipa, enforcement of password and account expiration is done by the server, not deferred to the clients. This is so we can do a DR test on some server or set of servers without also having to restore the IPA server first. I can do a getent passwd easily enough, and I could rebuild the shadow file for local users, so it's not critical, but it would be a nice to have in the case of a DR. What are you looking for in the shadow map ? 2) What is everyone else doing to prepare IPA for a DR? I've read that the best way to do it is to turn off the IPA services on a replica and then back that replica up. I also read that this will miss some important files that only exist on the master. This was true for ipa v1 only where we used a selfsigned CA available only in the first master, since v2 you are supposed to use the dogtag PKI, so if you clone the PKI as well (you need to explicitly set it up, by default replicas do not replicate the CA) you have full redundancy with regard to network facing data. I don't want to turn off the master server services for a DR due to failover lag. Would it be safe to take a backup of the master while hot, then restore a replica, and promote it to master using the hot backup of the master (just the specific CA files needed)? If you are using the dogtag CA it wouldn't as it uses a DS instance as well. If you are using the selfsigned CA well, I guess you have no other option. Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
