Re: [Freeipa-users] Question about removed replica, take two
Ludwig,
Thank you!
John DeSantis
2016-10-05 10:43 GMT-04:00 Ludwig Krispenz <lkris...@redhat.com>:
> Hi,
>
> the RUV in the replication agreement is maintained to control changelog
> trimming, no changes should be deleted from the changelog which have not
> been seen by all consumers. Since not always a connection for a replication
> agreement can be established, eg if the consumer is down, this information
> is made persistent and kept in the replication agreement.
> So, if you have references to removed servers in the agreement this should
> do no harm since teh changes have alredy be removed from the changelog
> during cleanallruv.
> The only scenario a problem could arise is if you reinstall a replica on one
> of the removed with a new replica ID, then you could end up with two replica
> ids with the same url and get the attrlist_replace errors.
>
> The removal of the replica id from the replication agreement RUV is noe
> handled by cleanallruv (upstream ticket #48414), but you can edit the
> dse.ldif and remove them manually
>
> Regards,
> Ludwig
>
>
> On 10/05/2016 03:07 PM, John Desantis wrote:
>>
>> Hello all (again),
>>
>> I think my reference to a disease prevented my message from being
>> delivered, despite seeing it posted on the list archive. I apologize
>> in advance for the additional "noise".
>>
>> Anyways, I was hoping some lingering questions could be answered
>> regarding some visible entries via ldapsearch, which manifest a
>> removed replica's hostname [1].
>>
>> Running the ipa-replica-manage and ipa-csreplica-manage commands do
>> not show the host in question any longer, but when I run a few
>> directory searches on each replica using the commands below:
>>
>> # ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
>> manager" -b "cn=config" objectclass=nsds5replica
>> # ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
>> manager" -b "cn=config" objectclass=nsds5replicationagreement
>>
>> I'm able to see the old host on the master, but not on the replicas. See
>> below.
>>
>> # master, replica id 4:
>> ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
>> manager" -b "cn=config" objectclass=nsds5replica|grep oldhost
>> nsDS5ReplicaBindDN:
>>
>> krbprincipalname=ldap/oldhost.dom.dom@dom.dom.dom,cn=services,cn=accounts,dc=dom,dc=dom,dc=dom
>>
>> ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
>> manager" -b "cn=config" objectclass=nsds5replicationagreement|grep
>> oldhost
>> nsds50ruv: {replica 24 ldap://oldhost.dom.dom.dom:389}
>> 5447f2520018 5447f8610018
>> nsruvReplicaLastModified: {replica 24 ldap://oldhost.dom.dom.dom:389}
>>
>> nsds50ruv: {replica 24 ldap://oldhost.dom.dom.dom:389}
>> 5447f2520018 5447f56b00020018
>> nsruvReplicaLastModified: {replica 24 ldap://oldhost.dom.dom.dom:389}
>>
>>
>> It's listed twice due to the other hosts in the topology.
>>
>> # replica id 22
>> ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
>> manager" -b "cn=config" objectclass=nsds5replica|grep oldhost
>> ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
>> manager" -b "cn=config" objectclass=nsds5replicationagreement|grep
>> oldhost
>>
>> # replica id 21
>> ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
>> manager" -b "cn=config" objectclass=nsds5replica|grep oldhost
>> ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
>> manager" -b "cn=config" objectclass=nsds5replicationagreement|grep
>> oldhost
>>
>> The other two replicas no longer have the reference to the old host
>> after the CLEANALLRUV and CLEANRUV tasks performed by ldapmodify. I
>> then read via [2] that the dse.ldif could be manually edited to remove
>> references, but I'm not sure if that should be done if the general
>> opinion is that the old references aren't going to cause a problem.
>> Based upon the information above, is having a reference to the hold
>> host via the ldapsearch outputs above going to be a problem? If the
>> entry shouldn't be there, should the ldapmodify be performed against
>> the
>> "cn=meTomaster.dom.dom.dom,cn=replica,cn=dc\3Ddom\2Cdc\3Ddom\2Cdc\3Ddom,cn=mapping
>> tree,cn=config" bases?
>>
>> For reference, these are the commands I ran to
[Freeipa-users] Question about removed replica, take two
Hello all (again),
I think my reference to a disease prevented my message from being
delivered, despite seeing it posted on the list archive. I apologize
in advance for the additional "noise".
Anyways, I was hoping some lingering questions could be answered
regarding some visible entries via ldapsearch, which manifest a
removed replica's hostname [1].
Running the ipa-replica-manage and ipa-csreplica-manage commands do
not show the host in question any longer, but when I run a few
directory searches on each replica using the commands below:
# ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
manager" -b "cn=config" objectclass=nsds5replica
# ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
manager" -b "cn=config" objectclass=nsds5replicationagreement
I'm able to see the old host on the master, but not on the replicas. See below.
# master, replica id 4:
ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
manager" -b "cn=config" objectclass=nsds5replica|grep oldhost
nsDS5ReplicaBindDN:
krbprincipalname=ldap/oldhost.dom.dom@dom.dom.dom,cn=services,cn=accounts,dc=dom,dc=dom,dc=dom
ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
manager" -b "cn=config" objectclass=nsds5replicationagreement|grep
oldhost
nsds50ruv: {replica 24 ldap://oldhost.dom.dom.dom:389}
5447f2520018 5447f8610018
nsruvReplicaLastModified: {replica 24 ldap://oldhost.dom.dom.dom:389}
nsds50ruv: {replica 24 ldap://oldhost.dom.dom.dom:389}
5447f2520018 5447f56b00020018
nsruvReplicaLastModified: {replica 24 ldap://oldhost.dom.dom.dom:389}
It's listed twice due to the other hosts in the topology.
# replica id 22
ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
manager" -b "cn=config" objectclass=nsds5replica|grep oldhost
ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
manager" -b "cn=config" objectclass=nsds5replicationagreement|grep
oldhost
# replica id 21
ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
manager" -b "cn=config" objectclass=nsds5replica|grep oldhost
ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
manager" -b "cn=config" objectclass=nsds5replicationagreement|grep
oldhost
The other two replicas no longer have the reference to the old host
after the CLEANALLRUV and CLEANRUV tasks performed by ldapmodify. I
then read via [2] that the dse.ldif could be manually edited to remove
references, but I'm not sure if that should be done if the general
opinion is that the old references aren't going to cause a problem.
Based upon the information above, is having a reference to the hold
host via the ldapsearch outputs above going to be a problem? If the
entry shouldn't be there, should the ldapmodify be performed against
the
"cn=meTomaster.dom.dom.dom,cn=replica,cn=dc\3Ddom\2Cdc\3Ddom\2Cdc\3Ddom,cn=mapping
tree,cn=config" bases?
For reference, these are the commands I ran to get to state [1]:
# master
ldapmodify -x -W -h localhost -D "cn=directory manager" <https://www.redhat.com/archives/freeipa-users/2016-August/msg00331.html
[2] https://www.redhat.com/archives/freeipa-users/2015-June/msg00382.html
Thanks!
John DeSantis
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
[Freeipa-users] Question about removed replica
Hello all,
Like a case of herpes, I'm back!
Anyways, I was hoping some lingering questions could be answered
regarding some visible entries via ldapsearch, which manifest a
removed replica's hostname [1].
Running the ipa-replica-manage and ipa-csreplica-manage commands do
not show the host in question any longer, but when I run a few
directory searches on each replica using the commands below:
# ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
manager" -b "cn=config" objectclass=nsds5replica
# ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
manager" -b "cn=config" objectclass=nsds5replicationagreement
I'm able to see the old host on the master, but not on the replicas. See below.
# master, replica id 4:
ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
manager" -b "cn=config" objectclass=nsds5replica|grep oldhost
nsDS5ReplicaBindDN:
krbprincipalname=ldap/oldhost.dom.dom@dom.dom.dom,cn=services,cn=accounts,dc=dom,dc=dom,dc=dom
ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
manager" -b "cn=config" objectclass=nsds5replicationagreement|grep
oldhost
nsds50ruv: {replica 24 ldap://oldhost.dom.dom.dom:389}
5447f2520018 5447f8610018
nsruvReplicaLastModified: {replica 24 ldap://oldhost.dom.dom.dom:389}
nsds50ruv: {replica 24 ldap://oldhost.dom.dom.dom:389}
5447f2520018 5447f56b00020018
nsruvReplicaLastModified: {replica 24 ldap://oldhost.dom.dom.dom:389}
It's listed twice due to the other hosts in the topology.
# replica id 22
ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
manager" -b "cn=config" objectclass=nsds5replica|grep oldhost
ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
manager" -b "cn=config" objectclass=nsds5replicationagreement|grep
oldhost
# replica id 21
ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
manager" -b "cn=config" objectclass=nsds5replica|grep oldhost
ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
manager" -b "cn=config" objectclass=nsds5replicationagreement|grep
oldhost
The other two replicas no longer have the reference to the old host
after the CLEANALLRUV and CLEANRUV tasks performed by ldapmodify. I
then read via [2] that the dse.ldif could be manually edited to remove
references, but I'm not sure if that should be done if the general
opinion is that the old references aren't going to cause a problem.
Based upon the information above, is having a reference to the hold
host via the ldapsearch outputs above going to be a problem? If the
entry shouldn't be there, should the ldapmodify be performed against
the
"cn=meTomaster.dom.dom.dom,cn=replica,cn=dc\3Ddom\2Cdc\3Ddom\2Cdc\3Ddom,cn=mapping
tree,cn=config" bases?
For reference, these are the commands I ran to get to state [1]:
# master
ldapmodify -x -W -h localhost -D "cn=directory manager" <https://www.redhat.com/archives/freeipa-users/2016-August/msg00331.html
[2] https://www.redhat.com/archives/freeipa-users/2015-June/msg00382.html
Thanks!
John DeSantis
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] replica_generate_next_csn messages in dirsrv error logs
Ludwig, > I looked into the logs, I think the messages are harmless, just an effect of > csn adjustment due to time difference on the two machines. I had said that > the replication protocol will try to adjust the csn generator, but looks > like you have long lasting replication connections and the adjustment is > done only at the beginning. Maybe we can look into this and improve it. > Just the tracking of one of these error messages: Thank you for your insight and looking into these logs. Given the relative obscurity of results of this message within Google and this mailing list, there may be nothing to improve! In other words, it looks like the issue has been resolved. In a not so nutshell, I've been monitoring a ns-slapd thread that was continually pegged with high file I/O via the 'pread' while reading its db* files on the master server, which produced some latency. After seemingly pointless searches, I stumbled upon Rich's "dbmon.sh" via a post [1] and verified that some tuning was needed for our site. After applying the changes I did notice that there was a drop in memory pressure on the system and that there seemed to be less latency, but the ns-slapd thread was still performing a lot of file I/O. It seems now that the issue with the timing was due to this observed latency. Anyways, I was still bugged with an issue I had originally opened in my first post to the list [2] and finally discovered that one of our replication nodes was culpable for not responding to the 'ipa-replica-manage clean-ruv #' (stdout via this command did not report which servers had and had not properly cleaned the RUV). I was able to manually remove it via 'ldapmodify' and cleanruv. At this point, some of the file I/O I had seen was more than halved. The last piece of the puzzle was using "ipa-csreplica-manage" and 'ldapmodify' to remove [3] the CA references to the replica mentioned in [1]. Once this was done, all of the thread I/O stopped. I then performed some testing of adding and removing DNS records via a for loop, with and without sleep statements. Not once did any more of the replica_generate_next_csn messages appear. For anyone else seeing similar issues, hopefully this information will help. John DeSantis [1] https://www.redhat.com/archives/freeipa-users/2014-November/msg00138.html [2] https://www.redhat.com/archives/freeipa-users/2014-October/msg00283.html [3] https://www.redhat.com/archives/freeipa-users/2015-March/msg00436.html 2016-08-22 5:41 GMT-04:00 Ludwig Krispenz <lkris...@redhat.com>: > Thanks, > > I looked into the logs, I think the messages are harmless, just an effect of > csn adjustment due to time difference on the two machines. I had said that > the replication protocol will try to adjust the csn generator, but looks > like you have long lasting replication connections and the adjustment is > done only at the beginning. Maybe we can look into this and improve it. > Just the tracking of one of these error messages: > > the entry is modified on adm3 > adm3 :[19/Aug/2016:15:47:05 -0400] conn=13 op=126637 MOD > dn="idnsname=rc.usf.edu,cn=dns,dc=rc,dc=usf,dc=edu" > adm3 :[19/Aug/2016:15:47:05 -0400] conn=13 op=126637 RESULT err=0 tag=103 > nentries=0 etime=0 csn=57b763030016 > this mod is replicated to adm0 > adm0 :[19/Aug/2016:15:47:06 -0400] conn=1395 op=86121 MOD > dn="idnsname=rc.usf.edu,cn=dns,dc=rc,dc=usf,dc=edu" > adm0 :[19/Aug/2016:15:47:06 -0400] conn=1395 op=86121 RESULT err=0 tag=103 > nentries=0 etime=0 csn=57b763030016 > the entry is modified again on adm0 > adm0 :[19/Aug/2016:15:47:07 -0400] conn=27 op=1108697 MOD > dn="idnsname=rc.usf.edu,cn=dns,dc=rc,dc=usf,dc=edu" > but it looks like the csn generated is smaller than the one already in the > entry, and it is adjusted > adm0 :[19/Aug/2016:15:47:07 -0400] - replica_generate_next_csn: > opcsn=57b76301000a0004 <= basecsn=57b763030016, adjusted > opcsn=57b7630300010004 > then the result is logged with the adjusted csn > adm0 :[19/Aug/2016:15:47:07 -0400] conn=27 op=1108697 RESULT err=0 tag=103 > nentries=0 etime=0 csn=57b7630300010004 > > so the mechanism works, but the messages may be confusing and improvement of > the protocol could be investigated. > > One question I have, but someone more familiar with dns should answer: > we have regular updates of the same entry on both replicas, about every 2 > seconds, what is the reason for this ? > > > /tmp/adm3-logs-del39-errors.txt:[19/Aug/2016:15:47:03 -0400] conn=13 > op=126630 MOD dn="idnsname=rc.usf.edu,cn=dns,dc=rc,dc=usf,dc=edu" > /tmp/adm3-logs-del39-errors.txt:[19/Aug/2016:15:47:05 -0400] conn=13 > op=126637 MOD dn="idnsname=rc.usf.edu,cn=dns,dc=rc,dc=usf,dc=edu" > /tmp/adm3-logs-del39-errors.txt:[19/Aug/2016:
Re: [Freeipa-users] replica_generate_next_csn messages in dirsrv error logs
Ludwig, > you still only grep the replication connection, but before being replicated > the entry has to be added by some client connection, can you get all > references to the entry ? > the log snippet you provide shows also csns with tag=103, which indicate a > MOD, are these MODs for the added entries ? or other mods ? . I can't believe I did that! Ok, so the logs have been rotated (I didn't think to adjust logrotate..), so there aren't any logs to peruse for the case I've presented so far. However, I was able to reproduce the errors by "bulk" deleting 39 DNS entries, and only the MASTER reported "replica_generate_next_csn" entries. Given the size of the logs, I think it would be pointless to do any kind of sanitization. I'll go ahead and gzip them for you and email you off-list. I've labeled them as MASTER and REPLICA. John DeSantis 2016-08-19 9:18 GMT-04:00 Ludwig Krispenz <lkris...@redhat.com>: > > On 08/18/2016 05:28 PM, John Desantis wrote: >> >> Ludwig, >> >>> unfortunately this is not enough to determine what is going on. The >>> intersting generated/used csn is only logged in the >>> corresponding RESULT message and these are only the replication >>> connections, >>> it would be necessary to see the >>> original ADD operation, was it added once or twice by a client ? >>> you could pick one entry eg server-6-3-sp and grep for all references in >>> the >>> access logs of both servers (maybe there are mods as well) and then >>> get also get the RESULT line for the ops found >> >> Here are the updated log snippets looking for ADD and RESULT: > > you still only grep the replication connection, but before being replicated > the entry has to be added by some client connection, can you get all > references to the entry ? > the log snippet you provide shows also csns with tag=103, which indicate a > MOD, are these MODs for the added entries ? or other mods ? > >> >> PROD:11:20:13-root@REPLICA:/var/log/dirsrv/slapd-DOM-DOM-DOM >> # grep -E '17/Aug/2016:13:50:4.*conn=602.*(RESULT|ADD)' access.2016081* >> access.20160817-124811:[17/Aug/2016:13:50:41 -0400] conn=602 op=4139 >> RESULT err=0 tag=120 nentries=0 etime=0 >> access.20160817-124811:[17/Aug/2016:13:50:41 -0400] conn=602 op=4140 >> RESULT err=0 tag=120 nentries=0 etime=0 >> access.20160817-124811:[17/Aug/2016:13:50:42 -0400] conn=602 op=4141 >> RESULT err=0 tag=120 nentries=0 etime=0 >> access.20160817-124811:[17/Aug/2016:13:50:42 -0400] conn=602 op=4142 >> RESULT err=0 tag=120 nentries=0 etime=0 >> access.20160817-124811:[17/Aug/2016:13:50:42 -0400] conn=602 op=4143 >> ADD >> dn="idnsname=server-6-3-sp,idnsname=dom.dom.dom,cn=dns,dc=dom,dc=dom,dc=dom" >> access.20160817-124811:[17/Aug/2016:13:50:42 -0400] conn=602 op=4143 >> RESULT err=0 tag=105 nentries=0 etime=0 csn=57b4a4bb00030004 >> access.20160817-124811:[17/Aug/2016:13:50:44 -0400] conn=602 op=4144 >> RESULT err=0 tag=103 nentries=0 etime=0 csn=57b4a4bb00040004 >> access.20160817-124811:[17/Aug/2016:13:50:44 -0400] conn=602 op=4145 >> RESULT err=0 tag=103 nentries=0 etime=0 >> access.20160817-124811:[17/Aug/2016:13:50:47 -0400] conn=602 op=4146 >> RESULT err=0 tag=120 nentries=0 etime=0 >> access.20160817-124811:[17/Aug/2016:13:50:47 -0400] conn=602 op=4147 >> RESULT err=0 tag=120 nentries=0 etime=0 >> access.20160817-124811:[17/Aug/2016:13:50:47 -0400] conn=602 op=4148 >> ADD >> dn="idnsname=server-6-4-sp,idnsname=dom.dom.dom,cn=dns,dc=dom,dc=dom,dc=dom" >> access.20160817-124811:[17/Aug/2016:13:50:47 -0400] conn=602 op=4148 >> RESULT err=0 tag=105 nentries=0 etime=0 csn=57b4a4bb00080004 >> access.20160817-124811:[17/Aug/2016:13:50:49 -0400] conn=602 op=4149 >> RESULT err=0 tag=103 nentries=0 etime=0 csn=57b4a4bc00010004 >> access.20160817-124811:[17/Aug/2016:13:50:49 -0400] conn=602 op=4150 >> RESULT err=0 tag=103 nentries=0 etime=0 >> access.20160817-124811:[17/Aug/2016:13:50:49 -0400] conn=602 op=4151 >> ADD >> dn="idnsname=server-6-5-sp,idnsname=dom.dom.dom,cn=dns,dc=dom,dc=dom,dc=dom" >> access.20160817-124811:[17/Aug/2016:13:50:49 -0400] conn=602 op=4151 >> RESULT err=0 tag=105 nentries=0 etime=0 csn=57b4a4c100050004 >> access.20160817-124811:[17/Aug/2016:13:50:49 -0400] conn=602 op=4152 >> RESULT err=0 tag=103 nentries=0 etime=0 csn=57b4a4c100060004 >> >> PROD:11:19:54-root@MASTER:/var/log/dirsrv/slapd-DOM-DOM-DOM >> # grep -E '17/Aug/2016:13:50:4.*conn=1395.*(RESULT|ADD)' access.2016081* >> access.20160817-111940:[17/Aug/2016:13:50:41 -0400] conn=1395 op=4148 >> RE
Re: [Freeipa-users] replica_generate_next_csn messages in dirsrv error logs
Ludwig, > unfortunately this is not enough to determine what is going on. The > intersting generated/used csn is only logged in the > corresponding RESULT message and these are only the replication connections, > it would be necessary to see the > original ADD operation, was it added once or twice by a client ? > you could pick one entry eg server-6-3-sp and grep for all references in the > access logs of both servers (maybe there are mods as well) and then > get also get the RESULT line for the ops found Here are the updated log snippets looking for ADD and RESULT: PROD:11:20:13-root@REPLICA:/var/log/dirsrv/slapd-DOM-DOM-DOM # grep -E '17/Aug/2016:13:50:4.*conn=602.*(RESULT|ADD)' access.2016081* access.20160817-124811:[17/Aug/2016:13:50:41 -0400] conn=602 op=4139 RESULT err=0 tag=120 nentries=0 etime=0 access.20160817-124811:[17/Aug/2016:13:50:41 -0400] conn=602 op=4140 RESULT err=0 tag=120 nentries=0 etime=0 access.20160817-124811:[17/Aug/2016:13:50:42 -0400] conn=602 op=4141 RESULT err=0 tag=120 nentries=0 etime=0 access.20160817-124811:[17/Aug/2016:13:50:42 -0400] conn=602 op=4142 RESULT err=0 tag=120 nentries=0 etime=0 access.20160817-124811:[17/Aug/2016:13:50:42 -0400] conn=602 op=4143 ADD dn="idnsname=server-6-3-sp,idnsname=dom.dom.dom,cn=dns,dc=dom,dc=dom,dc=dom" access.20160817-124811:[17/Aug/2016:13:50:42 -0400] conn=602 op=4143 RESULT err=0 tag=105 nentries=0 etime=0 csn=57b4a4bb00030004 access.20160817-124811:[17/Aug/2016:13:50:44 -0400] conn=602 op=4144 RESULT err=0 tag=103 nentries=0 etime=0 csn=57b4a4bb00040004 access.20160817-124811:[17/Aug/2016:13:50:44 -0400] conn=602 op=4145 RESULT err=0 tag=103 nentries=0 etime=0 access.20160817-124811:[17/Aug/2016:13:50:47 -0400] conn=602 op=4146 RESULT err=0 tag=120 nentries=0 etime=0 access.20160817-124811:[17/Aug/2016:13:50:47 -0400] conn=602 op=4147 RESULT err=0 tag=120 nentries=0 etime=0 access.20160817-124811:[17/Aug/2016:13:50:47 -0400] conn=602 op=4148 ADD dn="idnsname=server-6-4-sp,idnsname=dom.dom.dom,cn=dns,dc=dom,dc=dom,dc=dom" access.20160817-124811:[17/Aug/2016:13:50:47 -0400] conn=602 op=4148 RESULT err=0 tag=105 nentries=0 etime=0 csn=57b4a4bb00080004 access.20160817-124811:[17/Aug/2016:13:50:49 -0400] conn=602 op=4149 RESULT err=0 tag=103 nentries=0 etime=0 csn=57b4a4bc00010004 access.20160817-124811:[17/Aug/2016:13:50:49 -0400] conn=602 op=4150 RESULT err=0 tag=103 nentries=0 etime=0 access.20160817-124811:[17/Aug/2016:13:50:49 -0400] conn=602 op=4151 ADD dn="idnsname=server-6-5-sp,idnsname=dom.dom.dom,cn=dns,dc=dom,dc=dom,dc=dom" access.20160817-124811:[17/Aug/2016:13:50:49 -0400] conn=602 op=4151 RESULT err=0 tag=105 nentries=0 etime=0 csn=57b4a4c100050004 access.20160817-124811:[17/Aug/2016:13:50:49 -0400] conn=602 op=4152 RESULT err=0 tag=103 nentries=0 etime=0 csn=57b4a4c100060004 PROD:11:19:54-root@MASTER:/var/log/dirsrv/slapd-DOM-DOM-DOM # grep -E '17/Aug/2016:13:50:4.*conn=1395.*(RESULT|ADD)' access.2016081* access.20160817-111940:[17/Aug/2016:13:50:41 -0400] conn=1395 op=4148 RESULT err=0 tag=120 nentries=0 etime=0 access.20160817-111940:[17/Aug/2016:13:50:41 -0400] conn=1395 op=4149 RESULT err=0 tag=120 nentries=0 etime=0 access.20160817-111940:[17/Aug/2016:13:50:41 -0400] conn=1395 op=4150 RESULT err=0 tag=103 nentries=0 etime=0 csn=57b4a4b900050016 access.20160817-111940:[17/Aug/2016:13:50:43 -0400] conn=1395 op=4151 ADD dn="idnsname=server-6-3-sp,idnsname=dom.dom.dom,cn=dns,dc=dom,dc=dom,dc=dom" access.20160817-111940:[17/Aug/2016:13:50:43 -0400] conn=1395 op=4151 RESULT err=0 tag=105 nentries=0 etime=0 access.20160817-111940:[17/Aug/2016:13:50:44 -0400] conn=1395 op=4152 RESULT err=0 tag=103 nentries=0 etime=1 csn=57b4a4bc0016 access.20160817-111940:[17/Aug/2016:13:50:46 -0400] conn=1395 op=4153 RESULT err=0 tag=120 nentries=0 etime=0 access.20160817-111940:[17/Aug/2016:13:50:47 -0400] conn=1395 op=4154 RESULT err=0 tag=120 nentries=0 etime=0 access.20160817-111940:[17/Aug/2016:13:50:47 -0400] conn=1395 op=4155 RESULT err=0 tag=120 nentries=0 etime=0 access.20160817-111940:[17/Aug/2016:13:50:47 -0400] conn=1395 op=4156 RESULT err=0 tag=120 nentries=0 etime=0 access.20160817-111940:[17/Aug/2016:13:50:48 -0400] conn=1395 op=4157 RESULT err=0 tag=103 nentries=0 etime=1 csn=57b4a4c100010016 access.20160817-111940:[17/Aug/2016:13:50:49 -0400] conn=1395 op=4158 ADD dn="idnsname=server-6-5-sp,idnsname=dom.dom.dom,cn=dns,dc=dom,dc=dom,dc=dom" access.20160817-111940:[17/Aug/2016:13:50:49 -0400] conn=1395 op=4158 RESULT err=0 tag=105 nentries=0 etime=0 access.20160817-111940:[17/Aug/2016:13:50:49 -0400] conn=1395 op=4159 RESULT err=0 tag=103 nentries=0 etime=0 access.20160817-111940:[17/Aug/2016:13:50:49 -0400] conn=1395 op=4160 RESULT err=0 tag=103 nentries=0 etime=0 csn=57b4a4c30016 I'm positive that I was the only one performing DNS updates during this time, and I was only using 1 console.
Re: [Freeipa-users] replica_generate_next_csn messages in dirsrv error logs
Ludwig, Thank you for your response! > This is a normal scenario, but you could check if the simultaneous updates > on 4 and 16 are intentional. In regards to the simultaneous updates, the only items I have noted so far are: * The time sync between the master (4) and replica (16) was off by about 1-2 seconds, with the latter being ahead; * There are continual log entries referencing "replication-multimaster-extop" and "Netscape Replication End Session" in the dirsrv "access" logs, and during one of the manifestations of "replica_generate_next_csn", I found this: PROD:08:46:08-root@REPLICA:/var/log/dirsrv/slapd-DOM-DOM-DOM # grep -E '17/Aug/2016:13:50:4.*conn=602.*ADD' access.2016081* access.20160817-124811:[17/Aug/2016:13:50:42 -0400] conn=602 op=4143 ADD dn="idnsname=server-6-3-sp,idnsname=dom.dom.dom,cn=dns,dc=dom,dc=dom,dc=dom" access.20160817-124811:[17/Aug/2016:13:50:47 -0400] conn=602 op=4148 ADD dn="idnsname=server-6-4-sp,idnsname=dom.dom.dom,cn=dns,dc=dom,dc=dom,dc=dom" access.20160817-124811:[17/Aug/2016:13:50:49 -0400] conn=602 op=4151 ADD dn="idnsname=server-6-5-sp,idnsname=dom.dom.dom,cn=dns,dc=dom,dc=dom,dc=dom" PROD:08:47:44-root@MASTER:/var/log/dirsrv/slapd-DOM-DOM-DOM # grep -E '17/Aug/2016:13:50:4.*conn=1395.*ADD' access.2016081* access.20160817-111940:[17/Aug/2016:13:50:43 -0400] conn=1395 op=4151 ADD dn="idnsname=server-6-3-sp,idnsname=dom.dom.dom,cn=dns,dc=dom,dc=dom,dc=dom" access.20160817-111940:[17/Aug/2016:13:50:49 -0400] conn=1395 op=4158 ADD dn="idnsname=server-6-5-sp,idnsname=dom.dom.dom,cn=dns,dc=dom,dc=dom,dc=dom" It looks like the entries for server-6-3-sp and 6-5-sp were referenced twice. Do you think that the time being off by 1-2 seconds between the master and replica could be the issue? The connection 602 is the replication between the replica and master, and the connection 1395 is the replication between the master and replica. Since I know these operations were performed using the console via a for loop 'ipa dnsrecord-add dom.dom.dom server-6-$i-sp --a-rec=10.250.12.$i' on one of our login nodes, do you think that specifying an _srv_ record in the DOMAIN configuration with the address of the master server, e.g.: ipa_server = _srv_, MASTER.dom.dom.dom could be the issue (coupled with the time syncing)? I know that these questions are probably leaning more towards the 389ds team, so feel free to pass me over to them if need be. Again, thank you very much for responding! John DeSantis 2016-08-18 4:14 GMT-04:00 Ludwig Krispenz <lkris...@redhat.com>: > > On 08/17/2016 08:54 PM, John Desantis wrote: >> >> Hello all, >> >> We've been re-using old host names and IP addresses for a new >> deployment of nodes, and recently I've been seeing the messages pasted >> below in the slapd-DC.DC.DC "error" log on our nodes. >> >> [17/Aug/2016:10:30:30 -0400] - replica_generate_next_csn: >> opcsn=57b475cd00120004 <= basecsn=57b475cf0016, adjusted >> opcsn=57b475cf00010004 >> [17/Aug/2016:11:09:44 -0400] - replica_generate_next_csn: >> opcsn=57b47f020004 <= basecsn=57b47f020016, adjusted >> opcsn=57b47f030004 >> [17/Aug/2016:11:09:44 -0400] - replica_generate_next_csn: >> opcsn=57b47f040004 <= basecsn=57b47f040016, adjusted >> opcsn=57b47f050004 >> [17/Aug/2016:11:10:33 -0400] - replica_generate_next_csn: >> opcsn=57b47f2f0014 <= basecsn=57b47f320016, adjusted >> opcsn=57b47f330004 >> [17/Aug/2016:13:50:45 -0400] - replica_generate_next_csn: >> opcsn=57b4a4bb00090004 <= basecsn=57b4a4bc0016, adjusted >> opcsn=57b4a4bc00010004 >> [17/Aug/2016:13:52:54 -0400] - replica_generate_next_csn: >> opcsn=57b4a53e000a0004 <= basecsn=57b4a53f0016, adjusted >> opcsn=57b4a53f00010004 >> [17/Aug/2016:13:53:15 -0400] - replica_generate_next_csn: >> opcsn=57b4a55200070004 <= basecsn=57b4a5530016, adjusted >> opcsn=57b4a55300010004 >> [17/Aug/2016:13:53:32 -0400] - replica_generate_next_csn: >> opcsn=57b4a56200090004 <= basecsn=57b4a5640016, adjusted >> opcsn=57b4a56400010004 > > Each modification (add/del/mod) gets a csn assignged used in replication > update resolution. And each assigned csn has to newer than an existing one. > The messages you see is from code that double checks that the entry doesn't > have already a lareg csn - and adjusts it. > The logs indicate that entries are more or less concurrently updated on > replica 4 and 16, and the updates from16 are received while processing the > updates on 4. > This is a normal scenario, but you could check if the simultaneou
[Freeipa-users] replica_generate_next_csn messages in dirsrv error logs
Hello all, We've been re-using old host names and IP addresses for a new deployment of nodes, and recently I've been seeing the messages pasted below in the slapd-DC.DC.DC "error" log on our nodes. [17/Aug/2016:10:30:30 -0400] - replica_generate_next_csn: opcsn=57b475cd00120004 <= basecsn=57b475cf0016, adjusted opcsn=57b475cf00010004 [17/Aug/2016:11:09:44 -0400] - replica_generate_next_csn: opcsn=57b47f020004 <= basecsn=57b47f020016, adjusted opcsn=57b47f030004 [17/Aug/2016:11:09:44 -0400] - replica_generate_next_csn: opcsn=57b47f040004 <= basecsn=57b47f040016, adjusted opcsn=57b47f050004 [17/Aug/2016:11:10:33 -0400] - replica_generate_next_csn: opcsn=57b47f2f0014 <= basecsn=57b47f320016, adjusted opcsn=57b47f330004 [17/Aug/2016:13:50:45 -0400] - replica_generate_next_csn: opcsn=57b4a4bb00090004 <= basecsn=57b4a4bc0016, adjusted opcsn=57b4a4bc00010004 [17/Aug/2016:13:52:54 -0400] - replica_generate_next_csn: opcsn=57b4a53e000a0004 <= basecsn=57b4a53f0016, adjusted opcsn=57b4a53f00010004 [17/Aug/2016:13:53:15 -0400] - replica_generate_next_csn: opcsn=57b4a55200070004 <= basecsn=57b4a5530016, adjusted opcsn=57b4a55300010004 [17/Aug/2016:13:53:32 -0400] - replica_generate_next_csn: opcsn=57b4a56200090004 <= basecsn=57b4a5640016, adjusted opcsn=57b4a56400010004 They seem to only occur when updating DNS entries, whether on the console or via the GUI (tail -f'ing the log). A search in this mailing-list returns nothing, but a message is found on the 389-ds list [1]; it seems to suggest that the messages aren't fatal and are purely informational, yet if they are occurring constantly that there could be a problem with the replication algorithm and/or deployment. We're using ipa-server 3.0.0-47 and 389-ds 1.2.11.15-60. Nothing has changed on the deployment side of things, and I don't recall seeing this message before. I'm wondering if it's safe to disregard these messages due to the re-use of the entries, or if something else should be looked into. Thank you, John DeSantis [1] https://fedorahosted.org/389/ticket/47959 -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] Certificate expired/renew problems
Marc, I experienced a similar issue earlier this year. Try restarting certmonger after temporarily changing the date back on the master. In our case that service had failed miserably and it didn't allow FreeIPA to renew the certificates properly. Our replicas however were hit with a bug [1] during this process. We applied the patched code and followed the same process and all was well. John DeSantis [1] https://fedorahosted.org/freeipa/ticket/4064 2015-06-05 11:12 GMT-04:00 Marc Wiatrowski w...@iglass.net: hello, I've got a problem with expired certificates in my ipa/IdM setup. I believe the root issue to be from the fact that when everything was first setup about a year ago and everything was replicated from a first ipa server which no longer exists. There are currently 3 ipa servers but none of them are the original. Couple days ago I started getting errors similar to '(SSL_ERROR_EXPIRED_CERT_ALERT) SSL peer rejected your certificate as expired' through the web management interface. After investigating with 'getcert list' I found that several certificates expired at 2015-05-31 18:48:55 UTC. I found http://www.freeipa.org/page/Howto/Promote_CA_to_Renewal_and_CRL_Master and followed the procedure for ipa 4.0 and everything seemed to go as expected. However this did not fix my issue. With more searching it looked like once the certificates are expired the auto renew will not work. Finding https://www.freeipa.org/page/Howto/CA_Certificate_Renewal#Procedure_in_IPA_.3C_4.0 to try to manually renew I am stuck at the the beginning with 'Give the CSR to your external CA.' I don't believe we had our certificates externally signed. They are whatever the original install put in place. Setting the date back in time reeks havoc on our environment so I'm reluctant to leave it for to long. I can get what I believe is the original CSR from /etc/pki-ca/CS.cfg but unsure what to do next or if this is even the road I should be going down. Things seem to be working for the most part except trying to make updates. Any help on what to do next, somewhere else to look, or if I'm going in the right direction would be greatly appreciated. thanks, Marc Info: CentOS 6.5 with some current updates including ipa-server-3.0.0-42.el6.centos.i686 certmonger-0.75.13-1.el6.i686 $ getcert list-cas CA 'SelfSign': is-default: no ca-type: INTERNAL:SELF next-serial-number: 01 CA 'IPA': is-default: no ca-type: EXTERNAL helper-location: /usr/libexec/certmonger/ipa-submit CA 'certmaster': is-default: no ca-type: EXTERNAL helper-location: /usr/libexec/certmonger/certmaster-submit CA 'dogtag-ipa-renew-agent': is-default: no ca-type: EXTERNAL helper-location: /usr/libexec/certmonger/dogtag-ipa-renew-agent-submit CA 'local': is-default: no ca-type: EXTERNAL helper-location: /usr/libexec/certmonger/local-submit CA 'dogtag-ipa-retrieve-agent-submit': is-default: no ca-type: EXTERNAL helper-location: /usr/libexec/certmonger/dogtag-ipa-retrieve-agent-submit $ getcert list Number of certificates and requests being tracked: 9. Request ID '20131204194012': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/nssdb',nickname='Server-Cert',token='NSS Certificate DB' certificate: type=NSSDB,location='/etc/pki/nssdb',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=IGLASS.NET subject: CN=spider01o,O=IGLASS.NET expires: 2015-12-05 19:40:13 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: track: yes auto-renew: yes Request ID '20141114162346': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=IGLASS.NET subject: CN=spider01o.iglass.net,O=IGLASS.NET expires: 2016-11-14 16:22:37 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: track: yes auto-renew: yes Request ID '20141114162434': status: MONITORING ca-error: Internal error: no response to http://spider01o.iglass.net:9180/ca/ee/ca/profileSubmit?profileId=caServerCertserial_num=1073545218renewal=truexml=true;. stuck: no key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin='x' certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=IGLASS.NET subject: CN=spider01o.iglass.net,O=IGLASS.NET expires: 2016-11-03 16:24:27
[Freeipa-users] Questions about nsslapd-sizelimit
Hello all! I believe I may be falling victim to the nsslapd-sizelimit's default setting of 2,000. I've been wondering why some JSON calls to IPA (3.0.37, user_find) have been failing to show all user accounts in the results. Checking the FreeIPA admin UI, I can clearly find the users in question, but no matter what changes I set in the UI on the the console with search record limits and time limits, only 2,000 entries are ever returned. A final test this morning by adding an account via the UI did not augment the 2,000 entries returned in the user list; searching for the user on the console with 'ipa user-show y* --all' and via the search frame in the UI found the user. Looking over the documentation, it's stated that you can use the UI to update the limits. However, the limit is already set at 10,000 for the number of records to be returned, and the time limit is set at 60. The current dse.ldiff states that the nsslapd-sizelimit is 2,000. Is it possible that IPA isn't respecting this value since the constant number is 2,000? Is it safe to change this value via an ldapmodify? Thank you! John DeSantis -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] Questions about nsslapd-sizelimit
Rob, Thanks for your reply. My predecessor had wrote code to pull user entries from the realm in order to verify that: 1.) A home directory is created (if not already) and apply the correct ownership; 2.) A work directory (Lustre) is created (if not already) and apply the correct ownership. Given what you've said, I'll perform a work-around within the code to get a list of active users from a database table vs. the current method. John DeSantis 2015-05-04 9:53 GMT-04:00 Rob Crittenden rcrit...@redhat.com: John Desantis wrote: Hello all! I believe I may be falling victim to the nsslapd-sizelimit's default setting of 2,000. I've been wondering why some JSON calls to IPA (3.0.37, user_find) have been failing to show all user accounts in the results. Checking the FreeIPA admin UI, I can clearly find the users in question, but no matter what changes I set in the UI on the the console with search record limits and time limits, only 2,000 entries are ever returned. A final test this morning by adding an account via the UI did not augment the 2,000 entries returned in the user list; searching for the user on the console with 'ipa user-show y* --all' and via the search frame in the UI found the user. Looking over the documentation, it's stated that you can use the UI to update the limits. However, the limit is already set at 10,000 for the number of records to be returned, and the time limit is set at 60. The current dse.ldiff states that the nsslapd-sizelimit is 2,000. Is it possible that IPA isn't respecting this value since the constant number is 2,000? Is it safe to change this value via an ldapmodify? Thank you! John DeSantis Why do you need to return 2000 users at one time? IPA purposely limits the number of entries returned by default (100) specifically to discourage enumeration which is expensive. It is safe to modify this value using ldapmodify. Increasing the value is not recommended. rob -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] Confused with certificate renewal ipa-server-3.0.0.0-37.el6.x86_64
Hello all, I didn't reply to the list, so I'll forward in my response. The only remaining hiccup is now the replica's certmonger service keeps dying while failing to re-issue the ipaCert in /etc/httpd/alias. Log snippets are below: Jan 7 12:17:02 python: certmonger restarted httpd Jan 7 12:17:03 certmonger: Certificate named ipaCert in token NSS Certificate DB in database /etc/httpd/alias issued by CA and saved. Jan 7 12:17:08 certmonger: Certificate named ipaCert in token NSS Certificate DB in database /etc/httpd/alias is no longer valid. Jan 7 12:17:40 certmonger: Certificate named ipaCert in token NSS Certificate DB in database /etc/httpd/alias issued by CA but not saved. The IPA services are running and the machine can be accessed (queries issued, web GUI, etc.) Would anyone have an idea of why a replica would have issues renewing the ipaCert? CCing Jan to advise, he is the most experienced in this area. Would file corruption within the file of the Request ID in /var/lib/certmonger/request have anything to do with this? autorenew=1 monitor=1 ca_name=dogtag-ipa-retrieve-agent-submit ca_profile=ipaCert submitted=20141228050011 cert=ESC[?1034h-BEGIN CERTIFICATE- I checked a few other random client nodes (and the master) and none of them are showing this corruption in their requests. I attempted to fix the corruption (editing the file) and subsequently restart certmonger with no luck. Thanks, John DeSantis Thanks, John DeSantis 2015-01-08 13:26 GMT-05:00 John Desantis desan...@mail.usf.edu: Hello all, The only remaining hiccup is now the replica's certmonger service keeps dying while failing to re-issue the ipaCert in /etc/httpd/alias. Log snippets are below: Jan 7 12:17:02 python: certmonger restarted httpd Jan 7 12:17:03 certmonger: Certificate named ipaCert in token NSS Certificate DB in database /etc/httpd/alias issued by CA and saved. Jan 7 12:17:08 certmonger: Certificate named ipaCert in token NSS Certificate DB in database /etc/httpd/alias is no longer valid. Jan 7 12:17:40 certmonger: Certificate named ipaCert in token NSS Certificate DB in database /etc/httpd/alias issued by CA but not saved. The IPA services are running and the machine can be accessed (queries issued, web GUI, etc.) Would anyone have an idea of why a replica would have issues renewing the ipaCert? CCing Jan to advise, he is the most experienced in this area. Would file corruption within the file of the Request ID in /var/lib/certmonger/request have anything to do with this? autorenew=1 monitor=1 ca_name=dogtag-ipa-retrieve-agent-submit ca_profile=ipaCert submitted=20141228050011 cert=ESC[?1034h-BEGIN CERTIFICATE- I checked a few other random client nodes (and the master) and none of them are showing this corruption in their requests. I attempted to fix the corruption (editing the file) and subsequently restart certmonger with no luck. Thanks, John DeSantis 2015-01-08 8:10 GMT-05:00 Martin Kosek mko...@redhat.com: On 01/07/2015 06:43 PM, John Desantis wrote: Hello all, Just an update on this issue for anyone else who experiences a similar issue. It looks like the automatic renewal of the certificates failed on our master due the certmonger service being stuck. I stopped the service, stopped IPA services, and then reset the date to a few days prior to the expiration. I then (following a mailing list post) restarted IPA and then certmonger. At this point, I checked the status of the certificates and saw that they were changing. Only the Server-Cert in /etc/httpd/alias was complaining this time of not being able to contact the CA. Another certmonger service restart corrected the issue. I can now re-provision nodes accordingly! Ok, good to hear! The only remaining hiccup is now the replica's certmonger service keeps dying while failing to re-issue the ipaCert in /etc/httpd/alias. Log snippets are below: Jan 7 12:17:02 python: certmonger restarted httpd Jan 7 12:17:03 certmonger: Certificate named ipaCert in token NSS Certificate DB in database /etc/httpd/alias issued by CA and saved. Jan 7 12:17:08 certmonger: Certificate named ipaCert in token NSS Certificate DB in database /etc/httpd/alias is no longer valid. Jan 7 12:17:40 certmonger: Certificate named ipaCert in token NSS Certificate DB in database /etc/httpd/alias issued by CA but not saved. The IPA services are running and the machine can be accessed (queries issued, web GUI, etc.) Would anyone have an idea of why a replica would have issues renewing the ipaCert? CCing Jan to advise, he is the most experienced in this area. Thank you, John DeSantis 2015-01-06 15:50 GMT-05:00 John Desantis desan...@mail.usf.edu: Hello all, Looking at the various online documentation regarding certificate renewals: http://www.freeipa.org/page/Howto/CA_Certificate_Renewal#Procedure_in_IPA_.3C_4.0 http://www.freeipa.org
Re: [Freeipa-users] Confused with certificate renewal ipa-server-3.0.0.0-37.el6.x86_64
Hello all, Just an update on this issue for anyone else who experiences a similar issue. It looks like the automatic renewal of the certificates failed on our master due the certmonger service being stuck. I stopped the service, stopped IPA services, and then reset the date to a few days prior to the expiration. I then (following a mailing list post) restarted IPA and then certmonger. At this point, I checked the status of the certificates and saw that they were changing. Only the Server-Cert in /etc/httpd/alias was complaining this time of not being able to contact the CA. Another certmonger service restart corrected the issue. I can now re-provision nodes accordingly! The only remaining hiccup is now the replica's certmonger service keeps dying while failing to re-issue the ipaCert in /etc/httpd/alias. Log snippets are below: Jan 7 12:17:02 python: certmonger restarted httpd Jan 7 12:17:03 certmonger: Certificate named ipaCert in token NSS Certificate DB in database /etc/httpd/alias issued by CA and saved. Jan 7 12:17:08 certmonger: Certificate named ipaCert in token NSS Certificate DB in database /etc/httpd/alias is no longer valid. Jan 7 12:17:40 certmonger: Certificate named ipaCert in token NSS Certificate DB in database /etc/httpd/alias issued by CA but not saved. The IPA services are running and the machine can be accessed (queries issued, web GUI, etc.) Would anyone have an idea of why a replica would have issues renewing the ipaCert? Thank you, John DeSantis 2015-01-06 15:50 GMT-05:00 John Desantis desan...@mail.usf.edu: Hello all, Looking at the various online documentation regarding certificate renewals: http://www.freeipa.org/page/Howto/CA_Certificate_Renewal#Procedure_in_IPA_.3C_4.0 http://www.freeipa.org/page/Certmonger https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Identity_Management_Guide/cas.html I have to admit that I am completely confused on how to proceed given that the links above reference external CA's. The certificate was created in house (no external issuer) from what I can tell (openssl x509 -issuer and via IPA GUI). Thankfully(?), none of the certificates listed via 'getcert list' have a status of CA_UNREACHABLE, although all of them state NEED_CSR. I'll paste the contents below, sanitized of couse. # getcert list Number of certificates and requests being tracked: 8. Request ID '20130110185936': status: NEED_CSR stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE.COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-EXAMPLE.COM/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE.COM',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=ipa.example.com,O=EXAMPLE.COM expires: 2015-01-11 18:59:35 UTC eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv EXAMPLE.COM track: yes auto-renew: yes Request ID '20130110190008': status: NEED_CSR stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=ipa.example.com,O=EXAMPLE.COM expires: 2015-01-11 19:00:07 UTC eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: track: yes auto-renew: yes Request ID '20130110190034': status: NEED_CSR stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=ipa.example.com,O=EXAMPLE.COM expires: 2015-01-11 19:00:34 UTC eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_httpd track: yes auto-renew: yes Request ID '20130410022007': status: NEED_CSR stuck: no key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin='377154649534' certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=CA Audit,O=EXAMPLE.COM expires: 2014-12-31 18:58:42 UTC pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert auditSigningCert cert-pki-ca track: yes auto-renew: yes Request ID '20130410022008': status: NEED_CSR stuck: no key pair storage: type=NSSDB,location
[Freeipa-users] Confused with certificate renewal ipa-server-3.0.0.0-37.el6.x86_64
-31 18:59:24 UTC eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '20130410022011': status: NEED_CSR stuck: no key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin='377154649534' certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=ipa.example.com,O=EXAMPLE.COM expires: 2014-12-31 18:58:41 UTC eku: id-kp-serverAuth pre-save command: post-save command: track: yes auto-renew: yes This issue was manifest when I attempted to re-provision a client node. I'll paste the errors reported by Apache: [Tue Jan 06 14:14:47 2015] [error] Bad remote server certificate: -8181 [Tue Jan 06 14:14:47 2015] [error] SSL Library Error: -8181 Certificate has expired [Tue Jan 06 14:14:47 2015] [error] Re-negotiation handshake failed: Not accepted by client!? FWIW, all IPA services are running for now. Any guidance would certainly be appreciated! If more information is required, let me know and I'll paste it in a reply. Thank you, John DeSantis -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project
Re: [Freeipa-users] Attempting to re-provision previous replica
Hello again, I was just wondering if there was an update on this thread? Since it is just one machine having an issue, do you (Rob and Rich) think a re-initialization from the master on the affected host would clear the clog? I have left it alone since Mark was brought into the discussion. Thank you! John DeSantis 2014-10-23 9:34 GMT-04:00 Rich Megginson rmegg...@redhat.com: On 10/23/2014 07:01 AM, John Desantis wrote: Rob and Rich, ipa-replica-manage del should have cleaned things up. You can clear out old RUVs with ipa-replica-manage too via list-ruv and clean-ruv. You use list-ruv to get the id# to clean and clean-ruv to do the actual cleaning. I remember having previously tried this task, but it had failed on older RUV's which were not even active (the KDC was under some strain so ipa queries were timing out). However, I ran it again and have been able to delete all but 1 (it's still running) RUV referencing the previous replica. I'll report back once the tasks finishes or fails. The last RUV is stuck on another replica. It fails with the following error: [23/Oct/2014:08:55:09 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Initiating CleanAllRUV Task... [23/Oct/2014:08:55:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Retrieving maxcsn... [23/Oct/2014:08:55:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Found maxcsn (5447f8610018) [23/Oct/2014:08:55:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Cleaning rid (24)... [23/Oct/2014:08:55:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Waiting to process all the updates from the deleted replica... [23/Oct/2014:08:55:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Waiting for all the replicas to be online... [23/Oct/2014:08:55:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Waiting for all the replicas to receive all the deleted replica updates... [23/Oct/2014:08:55:11 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Replica maxcsn (5447f56b00020018) is not caught up with deleted replica's maxcsn(5447f8610018) [23/Oct/2014:08:55:11 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Replica not caught up (agmt=cn=meToiparepbackup.our.personal.domain (iparepbackup:389)) [23/Oct/2014:08:55:11 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Not all replicas caught up, retrying in 10 seconds [23/Oct/2014:08:55:23 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Replica maxcsn (5447f56b00020018) is not caught up with deleted replica's maxcsn(5447f8610018) [23/Oct/2014:08:55:23 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Replica not caught up (agmt=cn=meToiparepbackup.our.personal.domain (iparepbackup:389)) [23/Oct/2014:08:55:23 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Not all replicas caught up, retrying in 20 seconds I then abort the task since the retrying went up to 14400 seconds. Mark, do you know what is going on here? Would this be a simple re-initialization from the master on the host iparepbackup? Thanks, John DeSantis 2014-10-22 16:03 GMT-04:00 John Desantis desan...@mail.usf.edu: Rob and Rich, ipa-replica-manage del should have cleaned things up. You can clear out old RUVs with ipa-replica-manage too via list-ruv and clean-ruv. You use list-ruv to get the id# to clean and clean-ruv to do the actual cleaning. I remember having previously tried this task, but it had failed on older RUV's which were not even active (the KDC was under some strain so ipa queries were timing out). However, I ran it again and have been able to delete all but 1 (it's still running) RUV referencing the previous replica. I'll report back once the tasks finishes or fails. Thanks, John DeSantis 2014-10-22 15:49 GMT-04:00 Rob Crittenden rcrit...@redhat.com: Rich Megginson wrote: On 10/22/2014 12:55 PM, John Desantis wrote: Richard, You should remove the unused ruv elements. I'm not sure why they were not cleaned. You may have to use cleanallruv manually. https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/9.0/html/Administration_Guide/Managing_Replication-Solving_Common_Replication_Conflicts.html#cleanruv note - use the cleanallruv procedure, not cleanruv. I'll try that, thanks for the guidance. What is the real problem you have? Did replication stop working? Are you getting error messages? I cannot get the host to be a replica. Each time I run `ipa-replica-install replica-info-host-in-question.our.personal.domain.gpg' it fails. I had assumed it was due to the fact that the host was already a replica, but had to be taken offline due to a hard disk failing. The machine was re-provisioned after the new hard drive was installed. Ok. I don't know if we have a documented procedure for that case. I assumed that if you first ran ipa-replica-manage del, then ipa-replica-prepare, then ipa-replica-install, that would take care of that. ipa-replica-manage del should
Re: [Freeipa-users] Attempting to re-provision previous replica
Rob and Rich, ipa-replica-manage del should have cleaned things up. You can clear out old RUVs with ipa-replica-manage too via list-ruv and clean-ruv. You use list-ruv to get the id# to clean and clean-ruv to do the actual cleaning. I remember having previously tried this task, but it had failed on older RUV's which were not even active (the KDC was under some strain so ipa queries were timing out). However, I ran it again and have been able to delete all but 1 (it's still running) RUV referencing the previous replica. I'll report back once the tasks finishes or fails. The last RUV is stuck on another replica. It fails with the following error: [23/Oct/2014:08:55:09 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Initiating CleanAllRUV Task... [23/Oct/2014:08:55:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Retrieving maxcsn... [23/Oct/2014:08:55:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Found maxcsn (5447f8610018) [23/Oct/2014:08:55:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Cleaning rid (24)... [23/Oct/2014:08:55:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Waiting to process all the updates from the deleted replica... [23/Oct/2014:08:55:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Waiting for all the replicas to be online... [23/Oct/2014:08:55:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Waiting for all the replicas to receive all the deleted replica updates... [23/Oct/2014:08:55:11 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Replica maxcsn (5447f56b00020018) is not caught up with deleted replica's maxcsn(5447f8610018) [23/Oct/2014:08:55:11 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Replica not caught up (agmt=cn=meToiparepbackup.our.personal.domain (iparepbackup:389)) [23/Oct/2014:08:55:11 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Not all replicas caught up, retrying in 10 seconds [23/Oct/2014:08:55:23 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Replica maxcsn (5447f56b00020018) is not caught up with deleted replica's maxcsn(5447f8610018) [23/Oct/2014:08:55:23 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Replica not caught up (agmt=cn=meToiparepbackup.our.personal.domain (iparepbackup:389)) [23/Oct/2014:08:55:23 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Not all replicas caught up, retrying in 20 seconds I then abort the task since the retrying went up to 14400 seconds. Would this be a simple re-initialization from the master on the host iparepbackup? Thanks, John DeSantis 2014-10-22 16:03 GMT-04:00 John Desantis desan...@mail.usf.edu: Rob and Rich, ipa-replica-manage del should have cleaned things up. You can clear out old RUVs with ipa-replica-manage too via list-ruv and clean-ruv. You use list-ruv to get the id# to clean and clean-ruv to do the actual cleaning. I remember having previously tried this task, but it had failed on older RUV's which were not even active (the KDC was under some strain so ipa queries were timing out). However, I ran it again and have been able to delete all but 1 (it's still running) RUV referencing the previous replica. I'll report back once the tasks finishes or fails. Thanks, John DeSantis 2014-10-22 15:49 GMT-04:00 Rob Crittenden rcrit...@redhat.com: Rich Megginson wrote: On 10/22/2014 12:55 PM, John Desantis wrote: Richard, You should remove the unused ruv elements. I'm not sure why they were not cleaned. You may have to use cleanallruv manually. https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/9.0/html/Administration_Guide/Managing_Replication-Solving_Common_Replication_Conflicts.html#cleanruv note - use the cleanallruv procedure, not cleanruv. I'll try that, thanks for the guidance. What is the real problem you have? Did replication stop working? Are you getting error messages? I cannot get the host to be a replica. Each time I run `ipa-replica-install replica-info-host-in-question.our.personal.domain.gpg' it fails. I had assumed it was due to the fact that the host was already a replica, but had to be taken offline due to a hard disk failing. The machine was re-provisioned after the new hard drive was installed. Ok. I don't know if we have a documented procedure for that case. I assumed that if you first ran ipa-replica-manage del, then ipa-replica-prepare, then ipa-replica-install, that would take care of that. ipa-replica-manage del should have cleaned things up. You can clear out old RUVs with ipa-replica-manage too via list-ruv and clean-ruv. You use list-ruv to get the id# to clean and clean-ruv to do the actual cleaning. When I enabled extra debugging during the installation process, the initial error was that the dirsrv instance couldn't be started. I checked into this and found that there were missing files in /etc/dirsrv/slapd-BLAH directory. I was then able to start dirsrv after copying some schema files from another replica
[Freeipa-users] Attempting to re-provision previous replica
Hello all,
First and foremost, a big thank you! to the FreeIPA developers for a
great product!
Now, to the point!
We're trying to re-provision a previous replica using the standard
documentation via the Red Hat site:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Identity_Management_Guide/Setting_up_IPA_Replicas.html
However, we're running into errors during the import process. The
errors are varied and fail at random steps; there was an issue with
NTP or HTTP or LDAP, etc. This did not happen when we promoted a
separate node to become a replica.
We had previously removed the replica via `ipa-replica-manage del` and
ensured that no trace of it being a replica existed: removed DNS
records and verified that the host enrollment was not present. I did
not use the --force and --cleanup options.
When I check RUV's against the host in question, there are several. I
also queried the tombstones against the host and found two entries
which have valid hex time stamps; coincidentally, out of the 9
tombstone entries, 2 have nsds50ruv time stamps. I'll paste
sanitized output below:
# ldapsaerch -x -W -LLL -D cn=directory manager -b
dc=our,dc=personal,dc=domain '(objectclass=nsTombstone)'
Enter LDAP Password:
dn: nsuniqueid=---,dc=our,dc=personal,dc=domain
objectClass: top
objectClass: nsTombstone
objectClass: extensibleobject
nsds50ruv: {replicageneration} 50ef13ae0004
nsds50ruv: {replica 4 ldap://master.our.personal.domain:389}
5164d1470004 5447bda 800010004
nsds50ruv: {replica 22 ldap://separatenode.our.personal.domain:389}
54107f9f0016 54436b 250016
nsds50ruv: {replica 21 ldap://iparepbackup.our.personal.domain:389}
51b734de0015 51b7 34ef00020015
nsds50ruv: {replica 19
ldap://host-in-question.our.personal.domain:389} 510d56c900010013
510d82 be00020013
nsds50ruv: {replica 18 ldap://host-in-question.our.personal.domain:389}
nsds50ruv: {replica 17 ldap://host-in-question.our.personal.domain:389}
nsds50ruv: {replica 16 ldap://host-in-question.our.personal.domain:389}
nsds50ruv: {replica 15 ldap://host-in-question.our.personal.domain:389}
nsds50ruv: {replica 14 ldap://host-in-question.our.personal.domain:389}
nsds50ruv: {replica 13 ldap://host-in-question.our.personal.domain:389}
nsds50ruv: {replica 12 ldap://host-in-question.our.personal.domain:389}
nsds50ruv: {replica 23
ldap://host-in-question.our.personal.domain:389} 5418770200020017
541878 9a0017
dc: our
nsruvReplicaLastModified: {replica 4
ldap://master.our.personal.domain:389} 5447bce8
nsruvReplicaLastModified: {replica 22
ldap://separatenode.our.personal.domain:389} 54436a5e
nsruvReplicaLastModified: {replica 21
ldap://iparepbackup.our.personal.domain:389}
nsruvReplicaLastModified: {replica 19
ldap://host-in-question.our.personal.domain:389}
nsruvReplicaLastModified: {replica 18
ldap://host-in-question.our.personal.domain:389}
nsruvReplicaLastModified: {replica 17
ldap://host-in-question.our.personal.domain:389}
nsruvReplicaLastModified: {replica 16
ldap://host-in-question.our.personal.domain:389}
nsruvReplicaLastModified: {replica 15
ldap://host-in-question.our.personal.domain:389}
nsruvReplicaLastModified: {replica 14
ldap://host-in-question.our.personal.domain:389}
nsruvReplicaLastModified: {replica 13
ldap://host-in-question.our.personal.domain:389}
nsruvReplicaLastModified: {replica 12
ldap://host-in-question.our.personal.domain:389}
nsruvReplicaLastModified: {replica 23
ldap://host-in-question.our.personal.domain:389}
dn:
nsuniqueid=c08a2803-5b5a11e2-a527ce8b-8fa47d35,cn=host-in-question.our.personal.domain,cn=maste
rs,cn=ipa,cn=etc,dc=our,dc=personal,dc=domain
objectClass: top
objectClass: nsContainer
objectClass: nsTombstone
cn: host-in-question.our.personal.domain
nsParentUniqueId: e6fa9418-5b5711e2-a1a9825b-daf5b5b0
dn:
nsuniqueid=664c4383-6d6311e2-8db6e946-de27dd8d,cn=host-in-question.our.personal.domain,cn=maste
rs,cn=ipa,cn=etc,dc=our,dc=personal,dc=domain
objectClass: top
objectClass: nsContainer
objectClass: nsTombstone
cn: host-in-question.our.personal.domain
nsParentUniqueId: e6fa9418-5b5711e2-a1a9825b-daf5b5b0
As you can see, the host-in-question has many RUV's and of which two
appear to be active and two entries which I believe (pardon my
ignorance) possibly correlate with the active entries of the
host-in-question.
Do these two tombstone entries need to be deleted with ldapdelete
before we can re-provision host-in-question and add it back as a
replica?
Thank you,
John DeSantis
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go To http://freeipa.org for more info on the project
Re: [Freeipa-users] Attempting to re-provision previous replica
Richard,
You helped me before in #freeipa, so I appreciate the assistance again.
What version of 389 are you using?
rpm -q 389-ds-base
389-ds-base-1.2.11.15-34.el6_5
Thanks,
John DeSantis
2014-10-22 12:09 GMT-04:00 Rich Megginson rmegg...@redhat.com:
On 10/22/2014 09:42 AM, John Desantis wrote:
Hello all,
First and foremost, a big thank you! to the FreeIPA developers for a
great product!
Now, to the point!
We're trying to re-provision a previous replica using the standard
documentation via the Red Hat site:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Identity_Management_Guide/Setting_up_IPA_Replicas.html
However, we're running into errors during the import process. The
errors are varied and fail at random steps; there was an issue with
NTP or HTTP or LDAP, etc. This did not happen when we promoted a
separate node to become a replica.
We had previously removed the replica via `ipa-replica-manage del` and
ensured that no trace of it being a replica existed: removed DNS
records and verified that the host enrollment was not present. I did
not use the --force and --cleanup options.
What version of 389 are you using?
rpm -q 389-ds-base
When I check RUV's against the host in question, there are several. I
also queried the tombstones against the host and found two entries
which have valid hex time stamps; coincidentally, out of the 9
tombstone entries, 2 have nsds50ruv time stamps. I'll paste
sanitized output below:
# ldapsaerch -x -W -LLL -D cn=directory manager -b
dc=our,dc=personal,dc=domain '(objectclass=nsTombstone)'
Enter LDAP Password:
dn:
nsuniqueid=---,dc=our,dc=personal,dc=domain
objectClass: top
objectClass: nsTombstone
objectClass: extensibleobject
nsds50ruv: {replicageneration} 50ef13ae0004
nsds50ruv: {replica 4 ldap://master.our.personal.domain:389}
5164d1470004 5447bda 800010004
nsds50ruv: {replica 22 ldap://separatenode.our.personal.domain:389}
54107f9f0016 54436b 250016
nsds50ruv: {replica 21 ldap://iparepbackup.our.personal.domain:389}
51b734de0015 51b7 34ef00020015
nsds50ruv: {replica 19
ldap://host-in-question.our.personal.domain:389} 510d56c900010013
510d82 be00020013
nsds50ruv: {replica 18 ldap://host-in-question.our.personal.domain:389}
nsds50ruv: {replica 17 ldap://host-in-question.our.personal.domain:389}
nsds50ruv: {replica 16 ldap://host-in-question.our.personal.domain:389}
nsds50ruv: {replica 15 ldap://host-in-question.our.personal.domain:389}
nsds50ruv: {replica 14 ldap://host-in-question.our.personal.domain:389}
nsds50ruv: {replica 13 ldap://host-in-question.our.personal.domain:389}
nsds50ruv: {replica 12 ldap://host-in-question.our.personal.domain:389}
nsds50ruv: {replica 23
ldap://host-in-question.our.personal.domain:389} 5418770200020017
541878 9a0017
dc: our
nsruvReplicaLastModified: {replica 4
ldap://master.our.personal.domain:389} 5447bce8
nsruvReplicaLastModified: {replica 22
ldap://separatenode.our.personal.domain:389} 54436a5e
nsruvReplicaLastModified: {replica 21
ldap://iparepbackup.our.personal.domain:389}
nsruvReplicaLastModified: {replica 19
ldap://host-in-question.our.personal.domain:389}
nsruvReplicaLastModified: {replica 18
ldap://host-in-question.our.personal.domain:389}
nsruvReplicaLastModified: {replica 17
ldap://host-in-question.our.personal.domain:389}
nsruvReplicaLastModified: {replica 16
ldap://host-in-question.our.personal.domain:389}
nsruvReplicaLastModified: {replica 15
ldap://host-in-question.our.personal.domain:389}
nsruvReplicaLastModified: {replica 14
ldap://host-in-question.our.personal.domain:389}
nsruvReplicaLastModified: {replica 13
ldap://host-in-question.our.personal.domain:389}
nsruvReplicaLastModified: {replica 12
ldap://host-in-question.our.personal.domain:389}
nsruvReplicaLastModified: {replica 23
ldap://host-in-question.our.personal.domain:389}
dn:
nsuniqueid=c08a2803-5b5a11e2-a527ce8b-8fa47d35,cn=host-in-question.our.personal.domain,cn=maste
rs,cn=ipa,cn=etc,dc=our,dc=personal,dc=domain
objectClass: top
objectClass: nsContainer
objectClass: nsTombstone
cn: host-in-question.our.personal.domain
nsParentUniqueId: e6fa9418-5b5711e2-a1a9825b-daf5b5b0
dn:
nsuniqueid=664c4383-6d6311e2-8db6e946-de27dd8d,cn=host-in-question.our.personal.domain,cn=maste
rs,cn=ipa,cn=etc,dc=our,dc=personal,dc=domain
objectClass: top
objectClass: nsContainer
objectClass: nsTombstone
cn: host-in-question.our.personal.domain
nsParentUniqueId: e6fa9418-5b5711e2-a1a9825b-daf5b5b0
As you can see, the host-in-question has many RUV's and of which two
appear to be active and two entries which I believe (pardon my
ignorance) possibly correlate with the active entries of the
host-in-question.
Do these two tombstone entries need
Re: [Freeipa-users] Attempting to re-provision previous replica
Rob and Rich, ipa-replica-manage del should have cleaned things up. You can clear out old RUVs with ipa-replica-manage too via list-ruv and clean-ruv. You use list-ruv to get the id# to clean and clean-ruv to do the actual cleaning. I remember having previously tried this task, but it had failed on older RUV's which were not even active (the KDC was under some strain so ipa queries were timing out). However, I ran it again and have been able to delete all but 1 (it's still running) RUV referencing the previous replica. I'll report back once the tasks finishes or fails. Thanks, John DeSantis 2014-10-22 15:49 GMT-04:00 Rob Crittenden rcrit...@redhat.com: Rich Megginson wrote: On 10/22/2014 12:55 PM, John Desantis wrote: Richard, You should remove the unused ruv elements. I'm not sure why they were not cleaned. You may have to use cleanallruv manually. https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/9.0/html/Administration_Guide/Managing_Replication-Solving_Common_Replication_Conflicts.html#cleanruv note - use the cleanallruv procedure, not cleanruv. I'll try that, thanks for the guidance. What is the real problem you have? Did replication stop working? Are you getting error messages? I cannot get the host to be a replica. Each time I run `ipa-replica-install replica-info-host-in-question.our.personal.domain.gpg' it fails. I had assumed it was due to the fact that the host was already a replica, but had to be taken offline due to a hard disk failing. The machine was re-provisioned after the new hard drive was installed. Ok. I don't know if we have a documented procedure for that case. I assumed that if you first ran ipa-replica-manage del, then ipa-replica-prepare, then ipa-replica-install, that would take care of that. ipa-replica-manage del should have cleaned things up. You can clear out old RUVs with ipa-replica-manage too via list-ruv and clean-ruv. You use list-ruv to get the id# to clean and clean-ruv to do the actual cleaning. When I enabled extra debugging during the installation process, the initial error was that the dirsrv instance couldn't be started. I checked into this and found that there were missing files in /etc/dirsrv/slapd-BLAH directory. I was then able to start dirsrv after copying some schema files from another replica. The install did move forward but then failed with Apache and its IPA configuration. I performed several uninstalls and re-installs, and at one point I got error code 3 from ipa-replica-install, which is why I was thinking that the old RUV's and tombstones were to blame. It could be. I'm really not sure what the problem is at this point. I think we'd need to see ipareplica-install.log to know for sure. It could be the sort of thing where it fails early but doesn't kill the install, so the last error is a red herring. rob Thanks, John DeSantis 2014-10-22 12:51 GMT-04:00 Rich Megginson rmegg...@redhat.com: On 10/22/2014 10:31 AM, John Desantis wrote: Richard, You helped me before in #freeipa, so I appreciate the assistance again. What version of 389 are you using? rpm -q 389-ds-base 389-ds-base-1.2.11.15-34.el6_5 Thanks, John DeSantis 2014-10-22 12:09 GMT-04:00 Rich Megginson rmegg...@redhat.com: On 10/22/2014 09:42 AM, John Desantis wrote: Hello all, First and foremost, a big thank you! to the FreeIPA developers for a great product! Now, to the point! We're trying to re-provision a previous replica using the standard documentation via the Red Hat site: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Identity_Management_Guide/Setting_up_IPA_Replicas.html However, we're running into errors during the import process. The errors are varied and fail at random steps; there was an issue with NTP or HTTP or LDAP, etc. This did not happen when we promoted a separate node to become a replica. We had previously removed the replica via `ipa-replica-manage del` and ensured that no trace of it being a replica existed: removed DNS records and verified that the host enrollment was not present. I did not use the --force and --cleanup options. What version of 389 are you using? rpm -q 389-ds-base You should remove the unused ruv elements. I'm not sure why they were not cleaned. You may have to use cleanallruv manually. https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/9.0/html/Administration_Guide/Managing_Replication-Solving_Common_Replication_Conflicts.html#cleanruv note - use the cleanallruv procedure, not cleanruv. When I check RUV's against the host in question, there are several. I also queried the tombstones against the host and found two entries which have valid hex time stamps; coincidentally, out of the 9 tombstone entries, 2 have nsds50ruv time stamps. I'll paste sanitized output below: # ldapsaerch -x -W -LLL -D cn=directory manager -b dc=our,dc=personal,dc
