Re: [Freeipa-users] re-initialize replica
On Tue, Oct 06, 2015 at 10:22:44AM -0400, Rob Crittenden wrote:
> Andrew E. Bruno wrote:
> > On Tue, Oct 06, 2015 at 09:35:08AM -0400, Rob Crittenden wrote:
> >> Andrew E. Bruno wrote:
> >>> The replica is not showing up when running ipa-replica-manage list.
> >>>
> >>> # ipa-replica-manage list
> >>> srv-m14-32.cbls.ccr.buffalo.edu: master
> >>> srv-m14-31-02.cbls.ccr.buffalo.edu: master
> >>>
> >>>
> >>> However, still seeing the ruvs in ldapsearch:
> >>>
> >>> ldapsearch -Y GSSAPI -b "cn=mapping tree,cn=config"
> >>> objectClass=nsDS5ReplicationAgreement -LL
> >>>
> >>>
> >>> nsds50ruv: {replica 5 ldap://srv-m14-30.cbls.ccr.buffalo.edu:389}
> >>> 55afec6b
> >>> 0005 55b2aa6800020005
> >>>
> >>>
> >>> ..
> >>>
> >>> nsds50ruv: {replica 91 ldap://srv-m14-30.cbls.ccr.buffalo.edu:389}
> >>> 55afecb
> >>> 0005b 55b13e74005b
> >>>
> >>>
> >>> Should I clean these manually? or can I run: ipa-replica-manage clean-ruv
> >>> 5
> >>>
> >>> Thanks again for the all the help.
> >>>
> >>> --Andrew
> >>>
> >>>
> >>
> >> Note that the list of masters comes from entries in IPA, not from
> >> replication agreements.
> >>
> >> ipa-replica-manage list-ruv will show the RUV data in a simpler way.
> >>
> >> Yeah, I'd use clean-ruv to clean them up.
> >>
> >> rob
> >>
> >>
> >
> > I get an error trying to clean-ruv:
> >
> > # ipa-replica-manage clean-ruv 5
> > Replica ID 5 not found
> >
> > Can these safely be ignored? or will we hit problems when adding the
> > replica back in?
>
> ipa-replica-manage list-ruv will show you the current RUV list. If it
> isn't there then yeah, you're done.
>
> Having unused RUV in a master causes it to do unnecessary replication
> calculations.
>
> rob
Yes, list-ruv seems to show the correct RUV list.
# ipa-replica-manage list-ruv
srv-m14-32.cbls.ccr.buffalo.edu:389: 4
srv-m14-31-02.cbls.ccr.buffalo.edu:389: 3
It's just the ldapsearch that's showing repid 5 :
ldapsearch -Y GSSAPI -b "cn=mapping tree,cn=config"
objectClass=nsDS5ReplicationAgreement -LL
dn: cn=meTosrv-m14-31-02.cbls.ccr.buffalo.edu,cn=replica,cn=dc\3Dcbls\2Cdc\3Dc
cr\2Cdc\3Dbuffalo\2Cdc\3Dedu,cn=mapping tree,cn=config
cn: meTosrv-m14-31-02.cbls.ccr.buffalo.edu
objectClass: nsds5replicationagreement
objectClass: top
..
nsds50ruv: {replica 5 ldap://srv-m14-30.cbls.ccr.buffalo.edu:389} 55afec6b
0005 55b2aa6800020005
dn: cn=masterAgreement1-srv-m14-31-02.cbls.ccr.buffalo.edu-pki-tomcat,cn=repli
ca,cn=o\3Dipaca,cn=mapping tree,cn=config
objectClass: top
objectClass: nsds5replicationagreement
...
nsds50ruv: {replica 91 ldap://srv-m14-30.cbls.ccr.buffalo.edu:389} 55afecb
0005b 55b13e74005b
Last time we had a replicate fail we manually ran a cleanall ruv via ldapmodify
for the ipaca rid which wasn't properly deleted. However, this time we're
seeing the rid in both ipca dn and the replica dn?
Just want to be sure.. are you saying these can be safely ignored? and we can
trust that the list-ruv is correct (and not causing unnecessary replication
calculations). We plan on adding the failed replica back with the same name.
--Andrew
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] re-initialize replica
Andrew E. Bruno wrote:
> On Tue, Oct 06, 2015 at 09:35:08AM -0400, Rob Crittenden wrote:
>> Andrew E. Bruno wrote:
>>> The replica is not showing up when running ipa-replica-manage list.
>>>
>>> # ipa-replica-manage list
>>> srv-m14-32.cbls.ccr.buffalo.edu: master
>>> srv-m14-31-02.cbls.ccr.buffalo.edu: master
>>>
>>>
>>> However, still seeing the ruvs in ldapsearch:
>>>
>>> ldapsearch -Y GSSAPI -b "cn=mapping tree,cn=config"
>>> objectClass=nsDS5ReplicationAgreement -LL
>>>
>>>
>>> nsds50ruv: {replica 5 ldap://srv-m14-30.cbls.ccr.buffalo.edu:389}
>>> 55afec6b
>>> 0005 55b2aa6800020005
>>>
>>>
>>> ..
>>>
>>> nsds50ruv: {replica 91 ldap://srv-m14-30.cbls.ccr.buffalo.edu:389}
>>> 55afecb
>>> 0005b 55b13e74005b
>>>
>>>
>>> Should I clean these manually? or can I run: ipa-replica-manage clean-ruv 5
>>>
>>> Thanks again for the all the help.
>>>
>>> --Andrew
>>>
>>>
>>
>> Note that the list of masters comes from entries in IPA, not from
>> replication agreements.
>>
>> ipa-replica-manage list-ruv will show the RUV data in a simpler way.
>>
>> Yeah, I'd use clean-ruv to clean them up.
>>
>> rob
>>
>>
>
> I get an error trying to clean-ruv:
>
> # ipa-replica-manage clean-ruv 5
> Replica ID 5 not found
>
> Can these safely be ignored? or will we hit problems when adding the
> replica back in?
ipa-replica-manage list-ruv will show you the current RUV list. If it
isn't there then yeah, you're done.
Having unused RUV in a master causes it to do unnecessary replication
calculations.
rob
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] re-initialize replica
On 10/06/2015 01:13 PM, Andrew E. Bruno wrote:
On Tue, Oct 06, 2015 at 12:53:04PM -0400, Mark Reynolds wrote:
On 10/06/2015 10:30 AM, Andrew E. Bruno wrote:
On Tue, Oct 06, 2015 at 10:22:44AM -0400, Rob Crittenden wrote:
Andrew E. Bruno wrote:
On Tue, Oct 06, 2015 at 09:35:08AM -0400, Rob Crittenden wrote:
Andrew E. Bruno wrote:
The replica is not showing up when running ipa-replica-manage list.
# ipa-replica-manage list
srv-m14-32.cbls.ccr.buffalo.edu: master
srv-m14-31-02.cbls.ccr.buffalo.edu: master
However, still seeing the ruvs in ldapsearch:
ldapsearch -Y GSSAPI -b "cn=mapping tree,cn=config"
objectClass=nsDS5ReplicationAgreement -LL
nsds50ruv: {replica 5 ldap://srv-m14-30.cbls.ccr.buffalo.edu:389} 55afec6b
0005 55b2aa6800020005
..
nsds50ruv: {replica 91 ldap://srv-m14-30.cbls.ccr.buffalo.edu:389} 55afecb
0005b 55b13e74005b
Should I clean these manually? or can I run: ipa-replica-manage clean-ruv 5
Thanks again for the all the help.
--Andrew
Note that the list of masters comes from entries in IPA, not from
replication agreements.
ipa-replica-manage list-ruv will show the RUV data in a simpler way.
Yeah, I'd use clean-ruv to clean them up.
rob
I get an error trying to clean-ruv:
# ipa-replica-manage clean-ruv 5
Replica ID 5 not found
Can these safely be ignored? or will we hit problems when adding the
replica back in?
ipa-replica-manage list-ruv will show you the current RUV list. If it
isn't there then yeah, you're done.
Having unused RUV in a master causes it to do unnecessary replication
calculations.
rob
Yes, list-ruv seems to show the correct RUV list.
# ipa-replica-manage list-ruv
srv-m14-32.cbls.ccr.buffalo.edu:389: 4
srv-m14-31-02.cbls.ccr.buffalo.edu:389: 3
It's just the ldapsearch that's showing repid 5 :
ldapsearch -Y GSSAPI -b "cn=mapping tree,cn=config"
objectClass=nsDS5ReplicationAgreement -LL
I think this can be ignored sicne its on the repl agreement, and not the
backend.
What does this ldapsearch return:
replace -b with your suffix
ldapsearch -Y GSSAPI -b|"dc=example,dc=com"
'(&(nsuniqueid=---)(objectclass=nstombstone))'
nsds50ruv|
Mark
Here's the results of the above query:
dn: cn=replica,cn=dc\3Dcbls\2Cdc\3Dccr\2Cdc\3Dbuffalo\2Cdc\3Dedu,cn=mapping tr
ee,cn=config
nsds50ruv: {replicageneration} 55a955910004
nsds50ruv: {replica 4 ldap://srv-m14-32.cbls.ccr.buffalo.edu:389} 55a955fa
0004 561400b300070004
nsds50ruv: {replica 3 ldap://srv-m14-31-02.cbls.ccr.buffalo.edu:389} 55a955960
003 5613f7b500020003
nsds50ruv: {replica 5} 5600051d0015 5600051d0015
Still see that replica 5? Is that normal?
It's still present, and if you were having replication issues its
possible the changelog recreated that old replica ID (replica 5) after
it was cleaned. This changelog resurrection bug has been fixed
upstream- fyi.
So, you need to rerun cleanallruv. If the IPA CLI is not detecting the
replica id you are trying to delete, then you can run the 389-ds-base
cleanallruv.pl script and run it on the server with the old rid:
cleanallruv.pl -D "cn=directory manager" -w password -b
"dc=cbls,dc=ccr,dc=buffalo,dc=edu" -r 5
Wait a minute, and rerun that ldapsearch to see if the replica ID was
removed/cleaned.
Mark
Thanks!
--Andrew
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] re-initialize replica
On Tue, Oct 06, 2015 at 02:29:49PM -0400, Mark Reynolds wrote:
>
>
> On 10/06/2015 01:13 PM, Andrew E. Bruno wrote:
> >On Tue, Oct 06, 2015 at 12:53:04PM -0400, Mark Reynolds wrote:
> >>
> >>On 10/06/2015 10:30 AM, Andrew E. Bruno wrote:
> >>>On Tue, Oct 06, 2015 at 10:22:44AM -0400, Rob Crittenden wrote:
> Andrew E. Bruno wrote:
> >On Tue, Oct 06, 2015 at 09:35:08AM -0400, Rob Crittenden wrote:
> >>Andrew E. Bruno wrote:
> >>>The replica is not showing up when running ipa-replica-manage list.
> >>>
> >>> # ipa-replica-manage list
> >>> srv-m14-32.cbls.ccr.buffalo.edu: master
> >>> srv-m14-31-02.cbls.ccr.buffalo.edu: master
> >>>
> >>>
> >>>However, still seeing the ruvs in ldapsearch:
> >>>
> >>>ldapsearch -Y GSSAPI -b "cn=mapping tree,cn=config"
> >>>objectClass=nsDS5ReplicationAgreement -LL
> >>>
> >>>
> >>>nsds50ruv: {replica 5 ldap://srv-m14-30.cbls.ccr.buffalo.edu:389}
> >>>55afec6b
> >>> 0005 55b2aa6800020005
> >>>
> >>>
> >>>..
> >>>
> >>>nsds50ruv: {replica 91 ldap://srv-m14-30.cbls.ccr.buffalo.edu:389}
> >>>55afecb
> >>> 0005b 55b13e74005b
> >>>
> >>>
> >>>Should I clean these manually? or can I run: ipa-replica-manage
> >>>clean-ruv 5
> >>>
> >>>Thanks again for the all the help.
> >>>
> >>>--Andrew
> >>>
> >>>
> >>Note that the list of masters comes from entries in IPA, not from
> >>replication agreements.
> >>
> >>ipa-replica-manage list-ruv will show the RUV data in a simpler way.
> >>
> >>Yeah, I'd use clean-ruv to clean them up.
> >>
> >>rob
> >>
> >>
> >I get an error trying to clean-ruv:
> >
> > # ipa-replica-manage clean-ruv 5
> > Replica ID 5 not found
> >
> >Can these safely be ignored? or will we hit problems when adding the
> >replica back in?
> ipa-replica-manage list-ruv will show you the current RUV list. If it
> isn't there then yeah, you're done.
>
> Having unused RUV in a master causes it to do unnecessary replication
> calculations.
>
> rob
> >>>Yes, list-ruv seems to show the correct RUV list.
> >>>
> >>># ipa-replica-manage list-ruv
> >>>srv-m14-32.cbls.ccr.buffalo.edu:389: 4
> >>>srv-m14-31-02.cbls.ccr.buffalo.edu:389: 3
> >>>
> >>>It's just the ldapsearch that's showing repid 5 :
> >>>
> >>>ldapsearch -Y GSSAPI -b "cn=mapping tree,cn=config"
> >>>objectClass=nsDS5ReplicationAgreement -LL
> >>I think this can be ignored sicne its on the repl agreement, and not the
> >>backend.
> >>
> >>What does this ldapsearch return:
> >>
> >>replace -b with your suffix
> >>
> >>ldapsearch -Y GSSAPI -b|"dc=example,dc=com"
> >>'(&(nsuniqueid=---)(objectclass=nstombstone))'
> >>nsds50ruv|
> >>
> >>
> >>Mark
> >
> >Here's the results of the above query:
> >
> >
> >dn: cn=replica,cn=dc\3Dcbls\2Cdc\3Dccr\2Cdc\3Dbuffalo\2Cdc\3Dedu,cn=mapping
> >tr
> > ee,cn=config
> >nsds50ruv: {replicageneration} 55a955910004
> >nsds50ruv: {replica 4 ldap://srv-m14-32.cbls.ccr.buffalo.edu:389}
> >55a955fa
> > 0004 561400b300070004
> >nsds50ruv: {replica 3 ldap://srv-m14-31-02.cbls.ccr.buffalo.edu:389}
> >55a955960
> > 003 5613f7b500020003
> >nsds50ruv: {replica 5} 5600051d0015 5600051d0015
> >
> >
> >Still see that replica 5? Is that normal?
> It's still present, and if you were having replication issues its possible
> the changelog recreated that old replica ID (replica 5) after it was
> cleaned. This changelog resurrection bug has been fixed upstream- fyi.
>
> So, you need to rerun cleanallruv. If the IPA CLI is not detecting the
> replica id you are trying to delete, then you can run the 389-ds-base
> cleanallruv.pl script and run it on the server with the old rid:
>
> cleanallruv.pl -D "cn=directory manager" -w password -b
> "dc=cbls,dc=ccr,dc=buffalo,dc=edu" -r 5
>
> Wait a minute, and rerun that ldapsearch to see if the replica ID was
> removed/cleaned.
>
> Mark
Worked like a charm. cleanallruv.pl removed the replicate id and I
verified it was gone via the ldapsearch.
Thanks so much. Appreciate all the help.
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] re-initialize replica
On Mon, Oct 05, 2015 at 02:48:48PM -0400, Rob Crittenden wrote: > Andrew E. Bruno wrote: > > On Mon, Oct 05, 2015 at 12:40:42PM +0200, Martin Kosek wrote: > >> On 10/02/2015 06:00 PM, Andrew E. Bruno wrote: > >>> On Fri, Oct 02, 2015 at 09:56:47AM -0400, Andrew E. Bruno wrote: > What's the best way to re-initialize a replica? > > Suppose one of your replicas goes south.. is there a command to tell > that replicate to re-initialize from the first master (instead of > removing/re-adding the replica from the topology)? > >>> > >>> Found the command I was looking for: > >>>ipa-replica-manage re-initialize --from xxx > >>> > >>> However, one of our replicates is down and can't seem to re-initialize > >>> it. Starting ipa fails (via systemctl restart ipa): > >>> > >>> ipactl status > >>> Directory Service: RUNNING > >>> krb5kdc Service: STOPPED > >>> kadmin Service: STOPPED > >>> named Service: STOPPED > >>> ipa_memcached Service: STOPPED > >>> httpd Service: STOPPED > >>> pki-tomcatd Service: STOPPED > >>> ipa-otpd Service: STOPPED > >>> ipa: INFO: The ipactl command was successful > >>> > >>> > >>> Errors from the dirsrv show: > >>> > >>> : GSSAPI Error: Unspecified GSS failure. Minor code may provide more > >>> information (No Kerberos credentials available)) errno 0 (Success) > >>> [02/Oct/2015:11:45:05 -0400] slapi_ldap_bind - Error: could not perform > >>> interactive bind for id [] authentication mechanism [GSSAPI]: error -2 > >>> (Local error) > >>> [02/Oct/2015:11:50:05 -0400] set_krb5_creds - Could not get initial > >>> credentials for principal [ldap/server@realm] in keytab > >>> [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for > >>> requested realm) > >>> [02/Oct/2015:11:50:05 -0400] slapd_ldap_sasl_interactive_bind - Error: > >>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 > >>> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS > >>> failure. Minor code may provide more information (No Kerberos > >>> credentials available)) errno 0 (Success) > >>> [02/Oct/2015:11:50:05 -0400] slapi_ldap_bind - Error: could not perform > >>> interactive bind for id [] authentication mechanism [GSSAPI]: error -2 > >>> (Local error) > >>> > >>> > >>> Attempting to re-initialize fails: > >>> > >>> ipa-replica-manage re-initialize --from master > >>> Connection timed out. > >>> > >>> > >>> I verified time is in sync and DNS forward/reverse resolution is working. > >>> > >>> Any pointers on what else to try? > >>> > >>> Thanks! > >>> > >>> --Andrew > >> > >> Given that your Kerberos server instance is down, I would start > >> investigating > >> Kerberos logs to see why. > > > > > > So looks like the dirsrv service comes up but with GSS errors about kerb > > credentials. However, the rest of the services including the krb5kdc > > fail to come up. Errors from the kdc logs suggest DNS: > > DS complaining about GSS is somewhat normal during startup as it is a > bit noisy. The other errors suggest there is no data in the backend. An > ldapsearch would confirm that. > > > > > LOOKING_UP_CLIENT: DNS/replica@REALM Server error > > > > FreeIPA is configured to serve DNS and this replica resolves it's own > > DNS in /etc/resolv.conf (127.0.0.1) > > > > I tried pointing /etc/resolv.conf to another (good) replica and even > > tried adjusting /etc/krb5.conf to point at another kdc to try and get a > > ticket however it still tries to connect to the local kdc (which fails > > to start). > > > > I'm inclined to re-install this replica and start fresh. I'm curious if > > we can re-kickstart this host from a fresh os/freeipa install and run > > the ipa-replica-manage re-initialize --from master command. The replica > > will have the same name.. is this possible? Would we need to backup the > > /var/lib/ipa/replica-info-XXX.gpg file? > > It needs to have its own principal in order to re-initialize. It sounds > like it has nothing which is why replication is failing. > > I'd recommend generating a new replica file. There is no value in > re-using the old one and it could be harmful if the certificates are > expired. > > You'll need to delete all replication agreements this master had and > you'll need to use the --force option since it won't be accessible. When > you re-install the master it will get all the current data as part of > the setup so no need to re-initialize after that. I force removed the replica and still seeing the RUV's show up. # ipa-replica-manage -v --force del srv-m14-30.cbls.ccr.buffalo.edu >From the logs: [06/Oct/2015:07:43:47 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Initiating CleanAllRUV Task... [06/Oct/2015:07:43:47 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Retrieving maxcsn... [06/Oct/2015:07:43:47 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Found maxcsn (5600051d0015) [06/Oct/2015:07:43:47 -0400] NSMMReplicationPlugin - CleanAllRUV Task:
Re: [Freeipa-users] re-initialize replica
Andrew E. Bruno wrote: > On Mon, Oct 05, 2015 at 02:48:48PM -0400, Rob Crittenden wrote: >> Andrew E. Bruno wrote: >>> On Mon, Oct 05, 2015 at 12:40:42PM +0200, Martin Kosek wrote: On 10/02/2015 06:00 PM, Andrew E. Bruno wrote: > On Fri, Oct 02, 2015 at 09:56:47AM -0400, Andrew E. Bruno wrote: >> What's the best way to re-initialize a replica? >> >> Suppose one of your replicas goes south.. is there a command to tell >> that replicate to re-initialize from the first master (instead of >> removing/re-adding the replica from the topology)? > > Found the command I was looking for: >ipa-replica-manage re-initialize --from xxx > > However, one of our replicates is down and can't seem to re-initialize > it. Starting ipa fails (via systemctl restart ipa): > > ipactl status > Directory Service: RUNNING > krb5kdc Service: STOPPED > kadmin Service: STOPPED > named Service: STOPPED > ipa_memcached Service: STOPPED > httpd Service: STOPPED > pki-tomcatd Service: STOPPED > ipa-otpd Service: STOPPED > ipa: INFO: The ipactl command was successful > > > Errors from the dirsrv show: > > : GSSAPI Error: Unspecified GSS failure. Minor code may provide more > information (No Kerberos credentials available)) errno 0 (Success) > [02/Oct/2015:11:45:05 -0400] slapi_ldap_bind - Error: could not perform > interactive bind for id [] authentication mechanism [GSSAPI]: error -2 > (Local error) > [02/Oct/2015:11:50:05 -0400] set_krb5_creds - Could not get initial > credentials for principal [ldap/server@realm] in keytab > [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for > requested realm) > [02/Oct/2015:11:50:05 -0400] slapd_ldap_sasl_interactive_bind - Error: > could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 > (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS > failure. Minor code may provide more information (No Kerberos > credentials available)) errno 0 (Success) > [02/Oct/2015:11:50:05 -0400] slapi_ldap_bind - Error: could not perform > interactive bind for id [] authentication mechanism [GSSAPI]: error -2 > (Local error) > > > Attempting to re-initialize fails: > > ipa-replica-manage re-initialize --from master > Connection timed out. > > > I verified time is in sync and DNS forward/reverse resolution is working. > > Any pointers on what else to try? > > Thanks! > > --Andrew Given that your Kerberos server instance is down, I would start investigating Kerberos logs to see why. >>> >>> >>> So looks like the dirsrv service comes up but with GSS errors about kerb >>> credentials. However, the rest of the services including the krb5kdc >>> fail to come up. Errors from the kdc logs suggest DNS: >> >> DS complaining about GSS is somewhat normal during startup as it is a >> bit noisy. The other errors suggest there is no data in the backend. An >> ldapsearch would confirm that. >> >>> >>> LOOKING_UP_CLIENT: DNS/replica@REALM Server error >>> >>> FreeIPA is configured to serve DNS and this replica resolves it's own >>> DNS in /etc/resolv.conf (127.0.0.1) >>> >>> I tried pointing /etc/resolv.conf to another (good) replica and even >>> tried adjusting /etc/krb5.conf to point at another kdc to try and get a >>> ticket however it still tries to connect to the local kdc (which fails >>> to start). >>> >>> I'm inclined to re-install this replica and start fresh. I'm curious if >>> we can re-kickstart this host from a fresh os/freeipa install and run >>> the ipa-replica-manage re-initialize --from master command. The replica >>> will have the same name.. is this possible? Would we need to backup the >>> /var/lib/ipa/replica-info-XXX.gpg file? >> >> It needs to have its own principal in order to re-initialize. It sounds >> like it has nothing which is why replication is failing. >> >> I'd recommend generating a new replica file. There is no value in >> re-using the old one and it could be harmful if the certificates are >> expired. >> >> You'll need to delete all replication agreements this master had and >> you'll need to use the --force option since it won't be accessible. When >> you re-install the master it will get all the current data as part of >> the setup so no need to re-initialize after that. > > I force removed the replica and still seeing the RUV's show up. > > # ipa-replica-manage -v --force del srv-m14-30.cbls.ccr.buffalo.edu > > > From the logs: > > [06/Oct/2015:07:43:47 -0400] NSMMReplicationPlugin - CleanAllRUV Task: > Initiating CleanAllRUV Task... > [06/Oct/2015:07:43:47 -0400] NSMMReplicationPlugin - CleanAllRUV Task: > Retrieving maxcsn... > [06/Oct/2015:07:43:47 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Found > maxcsn (5600051d0015) >
Re: [Freeipa-users] re-initialize replica
On 10/06/2015 10:30 AM, Andrew E. Bruno wrote:
On Tue, Oct 06, 2015 at 10:22:44AM -0400, Rob Crittenden wrote:
Andrew E. Bruno wrote:
On Tue, Oct 06, 2015 at 09:35:08AM -0400, Rob Crittenden wrote:
Andrew E. Bruno wrote:
The replica is not showing up when running ipa-replica-manage list.
# ipa-replica-manage list
srv-m14-32.cbls.ccr.buffalo.edu: master
srv-m14-31-02.cbls.ccr.buffalo.edu: master
However, still seeing the ruvs in ldapsearch:
ldapsearch -Y GSSAPI -b "cn=mapping tree,cn=config"
objectClass=nsDS5ReplicationAgreement -LL
nsds50ruv: {replica 5 ldap://srv-m14-30.cbls.ccr.buffalo.edu:389} 55afec6b
0005 55b2aa6800020005
..
nsds50ruv: {replica 91 ldap://srv-m14-30.cbls.ccr.buffalo.edu:389} 55afecb
0005b 55b13e74005b
Should I clean these manually? or can I run: ipa-replica-manage clean-ruv 5
Thanks again for the all the help.
--Andrew
Note that the list of masters comes from entries in IPA, not from
replication agreements.
ipa-replica-manage list-ruv will show the RUV data in a simpler way.
Yeah, I'd use clean-ruv to clean them up.
rob
I get an error trying to clean-ruv:
# ipa-replica-manage clean-ruv 5
Replica ID 5 not found
Can these safely be ignored? or will we hit problems when adding the
replica back in?
ipa-replica-manage list-ruv will show you the current RUV list. If it
isn't there then yeah, you're done.
Having unused RUV in a master causes it to do unnecessary replication
calculations.
rob
Yes, list-ruv seems to show the correct RUV list.
# ipa-replica-manage list-ruv
srv-m14-32.cbls.ccr.buffalo.edu:389: 4
srv-m14-31-02.cbls.ccr.buffalo.edu:389: 3
It's just the ldapsearch that's showing repid 5 :
ldapsearch -Y GSSAPI -b "cn=mapping tree,cn=config"
objectClass=nsDS5ReplicationAgreement -LL
I think this can be ignored sicne its on the repl agreement, and not the
backend.
What does this ldapsearch return:
replace -b with your suffix
ldapsearch -Y GSSAPI -b|"dc=example,dc=com"
'(&(nsuniqueid=---)(objectclass=nstombstone))'
nsds50ruv|
Mark
dn: cn=meTosrv-m14-31-02.cbls.ccr.buffalo.edu,cn=replica,cn=dc\3Dcbls\2Cdc\3Dc
cr\2Cdc\3Dbuffalo\2Cdc\3Dedu,cn=mapping tree,cn=config
cn: meTosrv-m14-31-02.cbls.ccr.buffalo.edu
objectClass: nsds5replicationagreement
objectClass: top
..
nsds50ruv: {replica 5 ldap://srv-m14-30.cbls.ccr.buffalo.edu:389} 55afec6b
0005 55b2aa6800020005
dn: cn=masterAgreement1-srv-m14-31-02.cbls.ccr.buffalo.edu-pki-tomcat,cn=repli
ca,cn=o\3Dipaca,cn=mapping tree,cn=config
objectClass: top
objectClass: nsds5replicationagreement
...
nsds50ruv: {replica 91 ldap://srv-m14-30.cbls.ccr.buffalo.edu:389} 55afecb
0005b 55b13e74005b
Last time we had a replicate fail we manually ran a cleanall ruv via ldapmodify
for the ipaca rid which wasn't properly deleted. However, this time we're
seeing the rid in both ipca dn and the replica dn?
Just want to be sure.. are you saying these can be safely ignored? and we can
trust that the list-ruv is correct (and not causing unnecessary replication
calculations). We plan on adding the failed replica back with the same name.
--Andrew
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] re-initialize replica
On Tue, Oct 06, 2015 at 12:53:04PM -0400, Mark Reynolds wrote:
>
>
> On 10/06/2015 10:30 AM, Andrew E. Bruno wrote:
> >On Tue, Oct 06, 2015 at 10:22:44AM -0400, Rob Crittenden wrote:
> >>Andrew E. Bruno wrote:
> >>>On Tue, Oct 06, 2015 at 09:35:08AM -0400, Rob Crittenden wrote:
> Andrew E. Bruno wrote:
> >The replica is not showing up when running ipa-replica-manage list.
> >
> > # ipa-replica-manage list
> > srv-m14-32.cbls.ccr.buffalo.edu: master
> > srv-m14-31-02.cbls.ccr.buffalo.edu: master
> >
> >
> >However, still seeing the ruvs in ldapsearch:
> >
> >ldapsearch -Y GSSAPI -b "cn=mapping tree,cn=config"
> >objectClass=nsDS5ReplicationAgreement -LL
> >
> >
> >nsds50ruv: {replica 5 ldap://srv-m14-30.cbls.ccr.buffalo.edu:389}
> >55afec6b
> > 0005 55b2aa6800020005
> >
> >
> >..
> >
> >nsds50ruv: {replica 91 ldap://srv-m14-30.cbls.ccr.buffalo.edu:389}
> >55afecb
> > 0005b 55b13e74005b
> >
> >
> >Should I clean these manually? or can I run: ipa-replica-manage
> >clean-ruv 5
> >
> >Thanks again for the all the help.
> >
> >--Andrew
> >
> >
> Note that the list of masters comes from entries in IPA, not from
> replication agreements.
>
> ipa-replica-manage list-ruv will show the RUV data in a simpler way.
>
> Yeah, I'd use clean-ruv to clean them up.
>
> rob
>
>
> >>>I get an error trying to clean-ruv:
> >>>
> >>> # ipa-replica-manage clean-ruv 5
> >>> Replica ID 5 not found
> >>>
> >>>Can these safely be ignored? or will we hit problems when adding the
> >>>replica back in?
> >>ipa-replica-manage list-ruv will show you the current RUV list. If it
> >>isn't there then yeah, you're done.
> >>
> >>Having unused RUV in a master causes it to do unnecessary replication
> >>calculations.
> >>
> >>rob
> >Yes, list-ruv seems to show the correct RUV list.
> >
> ># ipa-replica-manage list-ruv
> >srv-m14-32.cbls.ccr.buffalo.edu:389: 4
> >srv-m14-31-02.cbls.ccr.buffalo.edu:389: 3
> >
> >It's just the ldapsearch that's showing repid 5 :
> >
> >ldapsearch -Y GSSAPI -b "cn=mapping tree,cn=config"
> >objectClass=nsDS5ReplicationAgreement -LL
> I think this can be ignored sicne its on the repl agreement, and not the
> backend.
>
> What does this ldapsearch return:
>
> replace -b with your suffix
>
> ldapsearch -Y GSSAPI -b|"dc=example,dc=com"
> '(&(nsuniqueid=---)(objectclass=nstombstone))'
> nsds50ruv|
>
>
> Mark
Here's the results of the above query:
dn: cn=replica,cn=dc\3Dcbls\2Cdc\3Dccr\2Cdc\3Dbuffalo\2Cdc\3Dedu,cn=mapping tr
ee,cn=config
nsds50ruv: {replicageneration} 55a955910004
nsds50ruv: {replica 4 ldap://srv-m14-32.cbls.ccr.buffalo.edu:389} 55a955fa
0004 561400b300070004
nsds50ruv: {replica 3 ldap://srv-m14-31-02.cbls.ccr.buffalo.edu:389} 55a955960
003 5613f7b500020003
nsds50ruv: {replica 5} 5600051d0015 5600051d0015
Still see that replica 5? Is that normal?
Thanks!
--Andrew
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] re-initialize replica
Andrew E. Bruno wrote: > On Mon, Oct 05, 2015 at 12:40:42PM +0200, Martin Kosek wrote: >> On 10/02/2015 06:00 PM, Andrew E. Bruno wrote: >>> On Fri, Oct 02, 2015 at 09:56:47AM -0400, Andrew E. Bruno wrote: What's the best way to re-initialize a replica? Suppose one of your replicas goes south.. is there a command to tell that replicate to re-initialize from the first master (instead of removing/re-adding the replica from the topology)? >>> >>> Found the command I was looking for: >>>ipa-replica-manage re-initialize --from xxx >>> >>> However, one of our replicates is down and can't seem to re-initialize >>> it. Starting ipa fails (via systemctl restart ipa): >>> >>> ipactl status >>> Directory Service: RUNNING >>> krb5kdc Service: STOPPED >>> kadmin Service: STOPPED >>> named Service: STOPPED >>> ipa_memcached Service: STOPPED >>> httpd Service: STOPPED >>> pki-tomcatd Service: STOPPED >>> ipa-otpd Service: STOPPED >>> ipa: INFO: The ipactl command was successful >>> >>> >>> Errors from the dirsrv show: >>> >>> : GSSAPI Error: Unspecified GSS failure. Minor code may provide more >>> information (No Kerberos credentials available)) errno 0 (Success) >>> [02/Oct/2015:11:45:05 -0400] slapi_ldap_bind - Error: could not perform >>> interactive bind for id [] authentication mechanism [GSSAPI]: error -2 >>> (Local error) >>> [02/Oct/2015:11:50:05 -0400] set_krb5_creds - Could not get initial >>> credentials for principal [ldap/server@realm] in keytab >>> [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for >>> requested realm) >>> [02/Oct/2015:11:50:05 -0400] slapd_ldap_sasl_interactive_bind - Error: >>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 >>> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS >>> failure. Minor code may provide more information (No Kerberos credentials >>> available)) errno 0 (Success) >>> [02/Oct/2015:11:50:05 -0400] slapi_ldap_bind - Error: could not perform >>> interactive bind for id [] authentication mechanism [GSSAPI]: error -2 >>> (Local error) >>> >>> >>> Attempting to re-initialize fails: >>> >>> ipa-replica-manage re-initialize --from master >>> Connection timed out. >>> >>> >>> I verified time is in sync and DNS forward/reverse resolution is working. >>> >>> Any pointers on what else to try? >>> >>> Thanks! >>> >>> --Andrew >> >> Given that your Kerberos server instance is down, I would start investigating >> Kerberos logs to see why. > > > So looks like the dirsrv service comes up but with GSS errors about kerb > credentials. However, the rest of the services including the krb5kdc > fail to come up. Errors from the kdc logs suggest DNS: DS complaining about GSS is somewhat normal during startup as it is a bit noisy. The other errors suggest there is no data in the backend. An ldapsearch would confirm that. > > LOOKING_UP_CLIENT: DNS/replica@REALM Server error > > FreeIPA is configured to serve DNS and this replica resolves it's own > DNS in /etc/resolv.conf (127.0.0.1) > > I tried pointing /etc/resolv.conf to another (good) replica and even > tried adjusting /etc/krb5.conf to point at another kdc to try and get a > ticket however it still tries to connect to the local kdc (which fails > to start). > > I'm inclined to re-install this replica and start fresh. I'm curious if > we can re-kickstart this host from a fresh os/freeipa install and run > the ipa-replica-manage re-initialize --from master command. The replica > will have the same name.. is this possible? Would we need to backup the > /var/lib/ipa/replica-info-XXX.gpg file? It needs to have its own principal in order to re-initialize. It sounds like it has nothing which is why replication is failing. I'd recommend generating a new replica file. There is no value in re-using the old one and it could be harmful if the certificates are expired. You'll need to delete all replication agreements this master had and you'll need to use the --force option since it won't be accessible. When you re-install the master it will get all the current data as part of the setup so no need to re-initialize after that. rob -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] re-initialize replica
On Mon, Oct 05, 2015 at 12:40:42PM +0200, Martin Kosek wrote: > On 10/02/2015 06:00 PM, Andrew E. Bruno wrote: > > On Fri, Oct 02, 2015 at 09:56:47AM -0400, Andrew E. Bruno wrote: > >> What's the best way to re-initialize a replica? > >> > >> Suppose one of your replicas goes south.. is there a command to tell > >> that replicate to re-initialize from the first master (instead of > >> removing/re-adding the replica from the topology)? > > > > Found the command I was looking for: > >ipa-replica-manage re-initialize --from xxx > > > > However, one of our replicates is down and can't seem to re-initialize > > it. Starting ipa fails (via systemctl restart ipa): > > > > ipactl status > > Directory Service: RUNNING > > krb5kdc Service: STOPPED > > kadmin Service: STOPPED > > named Service: STOPPED > > ipa_memcached Service: STOPPED > > httpd Service: STOPPED > > pki-tomcatd Service: STOPPED > > ipa-otpd Service: STOPPED > > ipa: INFO: The ipactl command was successful > > > > > > Errors from the dirsrv show: > > > > : GSSAPI Error: Unspecified GSS failure. Minor code may provide more > > information (No Kerberos credentials available)) errno 0 (Success) > > [02/Oct/2015:11:45:05 -0400] slapi_ldap_bind - Error: could not perform > > interactive bind for id [] authentication mechanism [GSSAPI]: error -2 > > (Local error) > > [02/Oct/2015:11:50:05 -0400] set_krb5_creds - Could not get initial > > credentials for principal [ldap/server@realm] in keytab > > [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for > > requested realm) > > [02/Oct/2015:11:50:05 -0400] slapd_ldap_sasl_interactive_bind - Error: > > could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 > > (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS > > failure. Minor code may provide more information (No Kerberos credentials > > available)) errno 0 (Success) > > [02/Oct/2015:11:50:05 -0400] slapi_ldap_bind - Error: could not perform > > interactive bind for id [] authentication mechanism [GSSAPI]: error -2 > > (Local error) > > > > > > Attempting to re-initialize fails: > > > > ipa-replica-manage re-initialize --from master > > Connection timed out. > > > > > > I verified time is in sync and DNS forward/reverse resolution is working. > > > > Any pointers on what else to try? > > > > Thanks! > > > > --Andrew > > Given that your Kerberos server instance is down, I would start investigating > Kerberos logs to see why. So looks like the dirsrv service comes up but with GSS errors about kerb credentials. However, the rest of the services including the krb5kdc fail to come up. Errors from the kdc logs suggest DNS: LOOKING_UP_CLIENT: DNS/replica@REALM Server error FreeIPA is configured to serve DNS and this replica resolves it's own DNS in /etc/resolv.conf (127.0.0.1) I tried pointing /etc/resolv.conf to another (good) replica and even tried adjusting /etc/krb5.conf to point at another kdc to try and get a ticket however it still tries to connect to the local kdc (which fails to start). I'm inclined to re-install this replica and start fresh. I'm curious if we can re-kickstart this host from a fresh os/freeipa install and run the ipa-replica-manage re-initialize --from master command. The replica will have the same name.. is this possible? Would we need to backup the /var/lib/ipa/replica-info-XXX.gpg file? --Andrew -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] re-initialize replica
On 10/02/2015 06:00 PM, Andrew E. Bruno wrote: > On Fri, Oct 02, 2015 at 09:56:47AM -0400, Andrew E. Bruno wrote: >> What's the best way to re-initialize a replica? >> >> Suppose one of your replicas goes south.. is there a command to tell >> that replicate to re-initialize from the first master (instead of >> removing/re-adding the replica from the topology)? > > Found the command I was looking for: >ipa-replica-manage re-initialize --from xxx > > However, one of our replicates is down and can't seem to re-initialize > it. Starting ipa fails (via systemctl restart ipa): > > ipactl status > Directory Service: RUNNING > krb5kdc Service: STOPPED > kadmin Service: STOPPED > named Service: STOPPED > ipa_memcached Service: STOPPED > httpd Service: STOPPED > pki-tomcatd Service: STOPPED > ipa-otpd Service: STOPPED > ipa: INFO: The ipactl command was successful > > > Errors from the dirsrv show: > > : GSSAPI Error: Unspecified GSS failure. Minor code may provide more > information (No Kerberos credentials available)) errno 0 (Success) > [02/Oct/2015:11:45:05 -0400] slapi_ldap_bind - Error: could not perform > interactive bind for id [] authentication mechanism [GSSAPI]: error -2 (Local > error) > [02/Oct/2015:11:50:05 -0400] set_krb5_creds - Could not get initial > credentials for principal [ldap/server@realm] in keytab > [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for > requested realm) > [02/Oct/2015:11:50:05 -0400] slapd_ldap_sasl_interactive_bind - Error: could > not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local > error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. > Minor code may provide more information (No Kerberos credentials available)) > errno 0 (Success) > [02/Oct/2015:11:50:05 -0400] slapi_ldap_bind - Error: could not perform > interactive bind for id [] authentication mechanism [GSSAPI]: error -2 (Local > error) > > > Attempting to re-initialize fails: > > ipa-replica-manage re-initialize --from master > Connection timed out. > > > I verified time is in sync and DNS forward/reverse resolution is working. > > Any pointers on what else to try? > > Thanks! > > --Andrew Given that your Kerberos server instance is down, I would start investigating Kerberos logs to see why. -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
[Freeipa-users] re-initialize replica
What's the best way to re-initialize a replica? Suppose one of your replicas goes south.. is there a command to tell that replicate to re-initialize from the first master (instead of removing/re-adding the replica from the topology)? Thanks, --Andrew -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] re-initialize replica
On Fri, Oct 02, 2015 at 09:56:47AM -0400, Andrew E. Bruno wrote: > What's the best way to re-initialize a replica? > > Suppose one of your replicas goes south.. is there a command to tell > that replicate to re-initialize from the first master (instead of > removing/re-adding the replica from the topology)? Found the command I was looking for: ipa-replica-manage re-initialize --from xxx However, one of our replicates is down and can't seem to re-initialize it. Starting ipa fails (via systemctl restart ipa): ipactl status Directory Service: RUNNING krb5kdc Service: STOPPED kadmin Service: STOPPED named Service: STOPPED ipa_memcached Service: STOPPED httpd Service: STOPPED pki-tomcatd Service: STOPPED ipa-otpd Service: STOPPED ipa: INFO: The ipactl command was successful Errors from the dirsrv show: : GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)) errno 0 (Success) [02/Oct/2015:11:45:05 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -2 (Local error) [02/Oct/2015:11:50:05 -0400] set_krb5_creds - Could not get initial credentials for principal [ldap/server@realm] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) [02/Oct/2015:11:50:05 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)) errno 0 (Success) [02/Oct/2015:11:50:05 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -2 (Local error) Attempting to re-initialize fails: ipa-replica-manage re-initialize --from master Connection timed out. I verified time is in sync and DNS forward/reverse resolution is working. Any pointers on what else to try? Thanks! --Andrew -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
