Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
On 28.6.2016 20:21, Sean Hogan wrote: > Thanks Petr, > > Since the last recycle of the Host hosting the First Master it has been > stable for about a week now. Only thing I did was to spread out my > replication agreements. I had 8 replications hitting it but now have 4 > going to it and the other 4 to its backup replica with the first master and > the backup replica having an agreement. > > > Not sure that fixed it or not but it seems to be stable at this point and I > know the docs say no more than 4 replications agreements so maybe it was > the cause. Generally more replication agreements mean more load on the server. Many replication agreements should not cause problems by itself if the server has sufficient performance. Petr^2 Spacek > Sean Hogan > > > > > > > > From: Petr Spacek > To: Sean Hogan/Durham/IBM@IBMUS > Cc: freeipa-users@redhat.com > Date: 06/28/2016 10:24 AM > Subject: Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem > > > > On 22.6.2016 23:09, Sean Hogan wrote: >> SLAPD showing >> >> 22/Jun/2016:17:01:59 -0400] slapi_ldap_bind - Error: could not perform >> interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials) >> [22/Jun/2016:17:06:59 -0400] slapd_ldap_sasl_interactive_bind - Error: >> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 >> (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: >> gss_accept_sec_context) errno 0 (Success) >> >> >> where would these creds be and what ID? I am using SASL so I assume it > to >> be sasl_user DNS/FirstMaster.watson.local or something like that? > > These are in /etc/dirsrv/ds.keytab. > > I would start with > # klist -kt /etc/dirsrv/ds.keytab > and try to proceed with kinit etc. (very similarly to the bind-dyndb-ldap > how-to). > > I hope it helps. > > Petr^2 Spacek > > >> From: Sean Hogan/Durham/IBM@IBMUS >> To: Petr Spacek >> Cc: freeipa-users@redhat.com >> Date: 06/22/2016 08:36 AM >> Subject: Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade >> problem >> Sent by: freeipa-users-boun...@redhat.com >> >> >> >> Hi Peter... >> >> Yes. this has me doing loops in my head to /dev/null >> >> You are correct I could not complete the BIND steps... I did them > yesterday >> but did not post results as I wanted to stop bugging you all :) >> The initial credential section of that I could not complete nor can I get >> an keytab without it and I don't think I have an issue with cert versions >> (used the SASL section). The upgrade log from 3.47 to 3.50 on this one >> server did show an error with named though. >> >> I had the box powered down again last night after testing the BIND >> procedures... and its been up since then. Which makes we really not sure >> what is going on(DNS DOS from internal maybe? I get a lot of outside >> requests showing network unreachable and I don't forward to a outside > DNS). >> If it was a password/cert/cipher/file perm issue then I don't see how it >> can work at all after a reboot. >> >> I am thinking it needs a rebuild.. I have not done this on a First Master >> IPA is there anything I need to be take into consider with it being first >> master? Right now I have 8 IPAs all DNS, NTP and CAs on differ vlans but >> the first master is the fail back IPA(on the only vlan that can talk to > the >> others) in case there local vlan IPA dies. First Master is also the > master >> CA in the realm where everything is enrolled to originally. We then mod >> everything to point to the vlan IPA with the Firstmaster as secondary > with >> our vlan-specific scripts we run after ipa client install. >> >> With the box rebooted last night I am now getting normal functionality > but >> it prob wont last long as indicated from the past... >> >> Working >> [bob@FirstMaster ~]# kinit admin >> Password for admin@DOMAIN.LOCAL: >> Warning: Your password will expire in 6 days on Tue Jun 28 14:55:52 2016 >> [bob@FirstMaster ~]# >> >> I did post ldap logs in my first email though... will readd them to this >> and when it dies off again I will add more. >> >> >>> [20/Jun/2016:13:59:00 -0400] - Detected Disorderly Shutdown last time >>> Directory Server was running, recovering database. >>> [20/Jun/2016:13:59:01 -0400] schema-compat-plugin - warning: no entries >> set >>> up under cn=computer
Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
Thanks Petr,
Since the last recycle of the Host hosting the First Master it has been
stable for about a week now. Only thing I did was to spread out my
replication agreements. I had 8 replications hitting it but now have 4
going to it and the other 4 to its backup replica with the first master and
the backup replica having an agreement.
Not sure that fixed it or not but it seems to be stable at this point and I
know the docs say no more than 4 replications agreements so maybe it was
the cause.
Sean Hogan
From: Petr Spacek
To: Sean Hogan/Durham/IBM@IBMUS
Cc: freeipa-users@redhat.com
Date: 06/28/2016 10:24 AM
Subject:Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
On 22.6.2016 23:09, Sean Hogan wrote:
> SLAPD showing
>
> 22/Jun/2016:17:01:59 -0400] slapi_ldap_bind - Error: could not perform
> interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials)
> [22/Jun/2016:17:06:59 -0400] slapd_ldap_sasl_interactive_bind - Error:
> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49
> (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure:
> gss_accept_sec_context) errno 0 (Success)
>
>
> where would these creds be and what ID? I am using SASL so I assume it
to
> be sasl_user DNS/FirstMaster.watson.local or something like that?
These are in /etc/dirsrv/ds.keytab.
I would start with
# klist -kt /etc/dirsrv/ds.keytab
and try to proceed with kinit etc. (very similarly to the bind-dyndb-ldap
how-to).
I hope it helps.
Petr^2 Spacek
> From: Sean Hogan/Durham/IBM@IBMUS
> To:Petr Spacek
> Cc:freeipa-users@redhat.com
> Date: 06/22/2016 08:36 AM
> Subject: Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade
> problem
> Sent by: freeipa-users-boun...@redhat.com
>
>
>
> Hi Peter...
>
> Yes. this has me doing loops in my head to /dev/null
>
> You are correct I could not complete the BIND steps... I did them
yesterday
> but did not post results as I wanted to stop bugging you all :)
> The initial credential section of that I could not complete nor can I get
> an keytab without it and I don't think I have an issue with cert versions
> (used the SASL section). The upgrade log from 3.47 to 3.50 on this one
> server did show an error with named though.
>
> I had the box powered down again last night after testing the BIND
> procedures... and its been up since then. Which makes we really not sure
> what is going on(DNS DOS from internal maybe? I get a lot of outside
> requests showing network unreachable and I don't forward to a outside
DNS).
> If it was a password/cert/cipher/file perm issue then I don't see how it
> can work at all after a reboot.
>
> I am thinking it needs a rebuild.. I have not done this on a First Master
> IPA is there anything I need to be take into consider with it being first
> master? Right now I have 8 IPAs all DNS, NTP and CAs on differ vlans but
> the first master is the fail back IPA(on the only vlan that can talk to
the
> others) in case there local vlan IPA dies. First Master is also the
master
> CA in the realm where everything is enrolled to originally. We then mod
> everything to point to the vlan IPA with the Firstmaster as secondary
with
> our vlan-specific scripts we run after ipa client install.
>
> With the box rebooted last night I am now getting normal functionality
but
> it prob wont last long as indicated from the past...
>
> Working
> [bob@FirstMaster ~]# kinit admin
> Password for admin@DOMAIN.LOCAL:
> Warning: Your password will expire in 6 days on Tue Jun 28 14:55:52 2016
> [bob@FirstMaster ~]#
>
> I did post ldap logs in my first email though... will readd them to this
> and when it dies off again I will add more.
>
>
>> [20/Jun/2016:13:59:00 -0400] - Detected Disorderly Shutdown last time
>> Directory Server was running, recovering database.
>> [20/Jun/2016:13:59:01 -0400] schema-compat-plugin - warning: no entries
> set
>> up under cn=computers, cn=compat,dc=domain,dc=local
>> [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin - ruv_compare_ruv:
RUV
>> [database RUV] does not contain element [{replica 7}
55ca26a90007
>> 5688d8e60017] which is present in RUV [changelog max RUV]
>> [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin -
>> replica_check_for_data_reload: Warning: for replica dc=domain,dc=local
>> there were some differences between the changelog max RUV and the
> database
>> RUV. If there are obsolete elements in the database RUV, you should
> remove
>> them using the CLEANALLRUV task. If they are not obsolete, you should
> check
>> their status to see why there are no changes from those
Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
On 22.6.2016 23:09, Sean Hogan wrote:
> SLAPD showing
>
> 22/Jun/2016:17:01:59 -0400] slapi_ldap_bind - Error: could not perform
> interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials)
> [22/Jun/2016:17:06:59 -0400] slapd_ldap_sasl_interactive_bind - Error:
> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49
> (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure:
> gss_accept_sec_context) errno 0 (Success)
>
>
> where would these creds be and what ID? I am using SASL so I assume it to
> be sasl_user DNS/FirstMaster.watson.local or something like that?
These are in /etc/dirsrv/ds.keytab.
I would start with
# klist -kt /etc/dirsrv/ds.keytab
and try to proceed with kinit etc. (very similarly to the bind-dyndb-ldap
how-to).
I hope it helps.
Petr^2 Spacek
> From: Sean Hogan/Durham/IBM@IBMUS
> To: Petr Spacek
> Cc: freeipa-users@redhat.com
> Date: 06/22/2016 08:36 AM
> Subject: Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
> Sent by: freeipa-users-boun...@redhat.com
>
>
>
> Hi Peter...
>
> Yes. this has me doing loops in my head to /dev/null
>
> You are correct I could not complete the BIND steps... I did them yesterday
> but did not post results as I wanted to stop bugging you all :)
> The initial credential section of that I could not complete nor can I get
> an keytab without it and I don't think I have an issue with cert versions
> (used the SASL section). The upgrade log from 3.47 to 3.50 on this one
> server did show an error with named though.
>
> I had the box powered down again last night after testing the BIND
> procedures... and its been up since then. Which makes we really not sure
> what is going on(DNS DOS from internal maybe? I get a lot of outside
> requests showing network unreachable and I don't forward to a outside DNS).
> If it was a password/cert/cipher/file perm issue then I don't see how it
> can work at all after a reboot.
>
> I am thinking it needs a rebuild.. I have not done this on a First Master
> IPA is there anything I need to be take into consider with it being first
> master? Right now I have 8 IPAs all DNS, NTP and CAs on differ vlans but
> the first master is the fail back IPA(on the only vlan that can talk to the
> others) in case there local vlan IPA dies. First Master is also the master
> CA in the realm where everything is enrolled to originally. We then mod
> everything to point to the vlan IPA with the Firstmaster as secondary with
> our vlan-specific scripts we run after ipa client install.
>
> With the box rebooted last night I am now getting normal functionality but
> it prob wont last long as indicated from the past...
>
> Working
> [bob@FirstMaster ~]# kinit admin
> Password for admin@DOMAIN.LOCAL:
> Warning: Your password will expire in 6 days on Tue Jun 28 14:55:52 2016
> [bob@FirstMaster ~]#
>
> I did post ldap logs in my first email though... will readd them to this
> and when it dies off again I will add more.
>
>
>> [20/Jun/2016:13:59:00 -0400] - Detected Disorderly Shutdown last time
>> Directory Server was running, recovering database.
>> [20/Jun/2016:13:59:01 -0400] schema-compat-plugin - warning: no entries
> set
>> up under cn=computers, cn=compat,dc=domain,dc=local
>> [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin - ruv_compare_ruv: RUV
>> [database RUV] does not contain element [{replica 7} 55ca26a90007
>> 5688d8e60017] which is present in RUV [changelog max RUV]
>> [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin -
>> replica_check_for_data_reload: Warning: for replica dc=domain,dc=local
>> there were some differences between the changelog max RUV and the
> database
>> RUV. If there are obsolete elements in the database RUV, you should
> remove
>> them using the CLEANALLRUV task. If they are not obsolete, you should
> check
>> their status to see why there are no changes from those servers in the
>> changelog.
>> [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial
>> credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in
>> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
>> for requested realm)
>> [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial
>> credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in
>> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
>> for requested realm)
>> [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial
>> credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in
>> keytab [FILE:/etc/dirsrv/ds
Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
SLAPD showing
22/Jun/2016:17:01:59 -0400] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials)
[22/Jun/2016:17:06:59 -0400] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49
(Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure:
gss_accept_sec_context) errno 0 (Success)
where would these creds be and what ID? I am using SASL so I assume it to
be sasl_user DNS/FirstMaster.watson.local or something like that?
Sean Hogan
From: Sean Hogan/Durham/IBM@IBMUS
To: Petr Spacek
Cc: freeipa-users@redhat.com
Date: 06/22/2016 08:36 AM
Subject:Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
Sent by:freeipa-users-boun...@redhat.com
Hi Peter...
Yes. this has me doing loops in my head to /dev/null
You are correct I could not complete the BIND steps... I did them yesterday
but did not post results as I wanted to stop bugging you all :)
The initial credential section of that I could not complete nor can I get
an keytab without it and I don't think I have an issue with cert versions
(used the SASL section). The upgrade log from 3.47 to 3.50 on this one
server did show an error with named though.
I had the box powered down again last night after testing the BIND
procedures... and its been up since then. Which makes we really not sure
what is going on(DNS DOS from internal maybe? I get a lot of outside
requests showing network unreachable and I don't forward to a outside DNS).
If it was a password/cert/cipher/file perm issue then I don't see how it
can work at all after a reboot.
I am thinking it needs a rebuild.. I have not done this on a First Master
IPA is there anything I need to be take into consider with it being first
master? Right now I have 8 IPAs all DNS, NTP and CAs on differ vlans but
the first master is the fail back IPA(on the only vlan that can talk to the
others) in case there local vlan IPA dies. First Master is also the master
CA in the realm where everything is enrolled to originally. We then mod
everything to point to the vlan IPA with the Firstmaster as secondary with
our vlan-specific scripts we run after ipa client install.
With the box rebooted last night I am now getting normal functionality but
it prob wont last long as indicated from the past...
Working
[bob@FirstMaster ~]# kinit admin
Password for admin@DOMAIN.LOCAL:
Warning: Your password will expire in 6 days on Tue Jun 28 14:55:52 2016
[bob@FirstMaster ~]#
I did post ldap logs in my first email though... will readd them to this
and when it dies off again I will add more.
> [20/Jun/2016:13:59:00 -0400] - Detected Disorderly Shutdown last time
> Directory Server was running, recovering database.
> [20/Jun/2016:13:59:01 -0400] schema-compat-plugin - warning: no entries
set
> up under cn=computers, cn=compat,dc=domain,dc=local
> [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin - ruv_compare_ruv: RUV
> [database RUV] does not contain element [{replica 7} 55ca26a90007
> 5688d8e60017] which is present in RUV [changelog max RUV]
> [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin -
> replica_check_for_data_reload: Warning: for replica dc=domain,dc=local
> there were some differences between the changelog max RUV and the
database
> RUV. If there are obsolete elements in the database RUV, you should
remove
> them using the CLEANALLRUV task. If they are not obsolete, you should
check
> their status to see why there are no changes from those servers in the
> changelog.
> [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial
> credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in
> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
> for requested realm)
> [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial
> credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in
> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
> for requested realm)
> [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial
> credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in
> keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
> for requested realm)
> [20/Jun/2016:13:59:48 -0400] slapd_ldap_sasl_interactive_bind - Error:
> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2
> (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
> failure. Minor code may provide more information (Credentials cache file
> '/tmp/krb5cc_495' not found)) errno 2 (No such file or directory)
> [20/Jun/2016:13:59:48 -0400] slapi_ldap_bind - Error: could not perform
> interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
> [20/Jun/2016:13:59:48 -0400]
Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
rror) (SASL(-1): generic failure: > GSSAPI Error: Unspecified GSS failure. Minor code may provide more > information (Credentials cache file '/tmp/krb5cc_495' not found)) > [20/Jun/2016:13:59:48 -0400] set_krb5_creds - Could not get initial > credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in > keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC > for requested realm) > [20/Jun/2016:13:59:48 -0400] slapd_ldap_sasl_interactive_bind - Error: > could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 > (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS > failure. Minor code may provide more information (Credentials cache file > '/tmp/krb5cc_495' not found)) errno 0 (Success) > [20/Jun/2016:13:59:48 -0400] slapi_ldap_bind - Error: could not perform > interactive bind for id [] mech [GSSAPI]: error -2 (Local error) > [20/Jun/2016:13:59:48 -0400] NSMMReplicationPlugin - > agmt="cn=meToserver2.domain.local" (server2:389): Replication bind with > GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: > GSSAPI Error: Unspecified GSS failure. Minor code may provide more > information (Credentials cache file '/tmp/krb5cc_495' not found)) > [20/Jun/2016:13:59:51 -0400] NSMMReplicationPlugin - > agmt="cn=meToserver8.domain.local" (server8:389): Replication bind with > GSSAPI auth resumed > [20/Jun/2016:13:59:51 -0400] slapd_ldap_sasl_interactive_bind - Error: > could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 > (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: > gss_accept_sec_context) errno 0 (Success) > [20/Jun/2016:13:59:51 -0400] slapi_ldap_bind - Error: could not perform > interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials) > [20/Jun/2016:13:59:51 -0400] NSMMReplicationPlugin - > agmt="cn=meToserver3.domain.local" (server3:389): Replication bind with > GSSAPI auth failed: LDAP error 49 (Invalid credentials) (SASL(-13): > authentication failure: GSSAPI Failure: gss_accept_sec_context) > [20/Jun/2016:13:59:51 -0400] slapd_ldap_sasl_interactive_bind - Error: > could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 > (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS > failure. Minor code may provide more information (No credentials cache > found)) errno 2 (No such file or directory) > [20/Jun/2016:13:59:51 -0400] slapi_ldap_bind - Error: could not perform > interactive bind for id [] mech [GSSAPI]: error -2 (Local error) > [20/Jun/2016:13:59:51 -0400] slapd_ldap_sasl_interactive_bind - Error: > could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 > (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS > failure. Minor code may provide more information (No credentials cache > found)) errno 2 (No such file or directory) > [20/Jun/2016:13:59:51 -0400] slapi_ldap_bind - Error: could not perform > interactive bind for id [] mech [GSSAPI]: error -2 (Local error) > [20/Jun/2016:13:59:51 -0400] slapd_ldap_sasl_interactive_bind - Error: > could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 > (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS > failure. Minor code may provide more information (No credentials cache > found)) errno 2 (No such file or directory) > [20/Jun/2016:13:59:51 -0400] slapi_ldap_bind - Error: could not perform > interactive bind for id [] mech [GSSAPI]: error -2 (Local error) > [20/Jun/2016:13:59:51 -0400] slapd_ldap_sasl_interactive_bind - Error: > could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 > (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS > failure. Minor code may provide more information (No credentials cache > found)) errno 2 (No such file or directory) > [20/Jun/2016:13:59:51 -0400] slapi_ldap_bind - Error: could not perform > interactive bind for id [] mech [GSSAPI]: error -2 (Local error) > [20/Jun/2016:13:59:51 -0400] slapd_ldap_sasl_interactive_bind - Error: > could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 > (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS > failure. Minor code may provide more information (No credentials cache > found)) errno 2 (No such file or directory) > [20/Jun/2016:13:59:51 -0400] slapi_ldap_bind - Error: could not perform > interactive bind for id [] mech [GSSAPI]: error -2 (Local error) > [20/Jun/2016:13:59:57 -0400] slapd_ldap_sasl_interactive_bind - Error: > could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 > (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: > gss_acce
Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
On 22.6.2016 02:56, Sean Hogan wrote:
> More info
>
>
> Krb5 log is showing:
> Jun 21 20:42:47 Firstmaster.domain.local krb5kdc[2141](info): AS_REQ (4
> etypes {18 17 16 23}) 10.x.x.x: LOOKING_UP_CLIENT: admin@domain.LOCAL for
> krbtgt/DOMAIN.LOCAL@DOMAIN.LOCAL, Server error
Hello,
this is really fishy. I would bet that there is a problem with LDAP server and
DNS errors are just consequence of it.
I suspect that you will not be able to finish steps mentioned in
https://fedorahosted.org/bind-dyndb-ldap/wiki/BIND9/NamedCannotStart#a3.FailedtoinitcredentialsorFailedtogetinitialcredentialsDecryptintegritycheckfailedorClientscredentialshavebeenrevoked
If it is the case I would turn your attention to krb5kdc.log and LDAP server
logs in /var/log/dirsrv/*
There must be something wrong with the LDAP server.
Petr^2 Spacek
>
> [bob@Firstmaster etc]# kinit -v admin
> kinit: Credentials cache file '/tmp/krb5cc_0' not found while validating
> credentials
>
>
>
>
>
>
> Sean Hogan
>
>
>
>
>
>
> From: Sean Hogan/Durham/IBM
> To: freeipa-users
> Date: 06/21/2016 12:02 PM
> Subject: Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
>
>
> Has anyone seen these before?
>
>
>
> First Master IPA DNS logs show: Looks like the host names are getting the
> domain twice domain.local.domain.local
>
>
> client 10.x.x.x#58094: query failed (SERVFAIL) for
> server1.domain.local.domain.local/IN/ at query.c:6569
> timeout in ldap_pool_getconnection(): try to raise 'connections' parameter;
> potential deadlock?
> client 10.x.x.x#44147: query failed (SERVFAIL) for
> x.x.x.10.in-addr.arpa/IN/PTR at query.c:6569
> timeout in ldap_pool_getconnection(): try to raise 'connections' parameter;
> potential deadlock?
> client 10.x.x.x#56466: query failed (SERVFAIL) for
> x.x.x.10.in-addr.arpa/IN/PTR at query.c:6569
> timeout in ldap_pool_getconnection(): try to raise 'connections' parameter;
> potential deadlock?
> client 10.x.x.x53367: query failed (SERVFAIL) for
> server2.domain.local.domain.local/IN/A at query.c:6569
> timeout in ldap_pool_getconnection(): try to raise 'connections' parameter;
> potential deadlock?
> client 10.x.x.x#53367: query failed (SERVFAIL) for
> server2.domain.local.domain.local/IN/ at query.c:6569
>
>
>
> So enrolls are failing at this point when tyring to enroll to a replica:
>
> [bob@server1 log]# ipa-client-install –enable-dns-updates
> Discovery was successful!
> Hostname: server1.watson.local
> Realm: DOMAIN.LOCAL
> DNS Domain: domain.local
> IPA Server: ipareplica.domain.local
> BaseDN: dc=domain,dc=local
>
> Continue to configure the system with these values? [no]: yes
> User authorized to enroll computers: bob
> Synchronizing time with KDC...
> Password for bob@DOMAIN.LOCAL:
> Successfully retrieved CA cert
> Subject: CN=Certificate Authority,O=DOMAIN.LOCAL
> Issuer: CN=Certificate Authority,O=DOMAIN.LOCAL
> Valid From: Tue Jan 06 19:37:09 2015 UTC
> Valid Until: Sat Jan 06 19:37:09 2035 UTC
>
> Enrolled in IPA realm DOMAIN.LOCAL
> Attempting to get host TGT...
> Created /etc/ipa/default.conf
> New SSSD config will be created
> Configured sudoers in /etc/nsswitch.conf
> Configured /etc/sssd/sssd.conf
> Configured /etc/krb5.conf for IPA realm DOMAIN.LOCAL
> trying https://ipareplica.domain.local/ipa/xml
> Cannot connect to the server due to Kerberos error: Kerberos error:
> Kerberos error: ('Unspecified GSS failure. Minor code may provide more
> information', 851968)/('KDC returned error string: PROCESS_TGS',
> -1765328324)/. Trying with delegate=True
> trying https://ipareplica.domain.local/ipa/xml
> Second connect with delegate=True also failed: Kerberos error: Kerberos
> error: ('Unspecified GSS failure. Minor code may provide more
> information', 851968)/('KDC returned error string: PROCESS_TGS',
> -1765328324)/
> Cannot connect to the IPA server XML-RPC interface: Kerberos error:
> Kerberos error: ('Unspecified GSS failure. Minor code may provide more
> information', 851968)/('KDC returned error string: PROCESS_TGS',
> -1765328324)/
> Installation failed. Rolling back changes.
> Unenrolling client from IPA server
> Unenrolling host failed: Error obtaining initial credentials: Generic error
> (see e-text).
>
> Removing Kerberos service principals from /etc/krb5.keytab
> Disabling client Kerberos and LDAP configurations
> Redundant SSSD configuration file /etc/sssd/sssd.conf was moved
> to /etc/sssd/sssd.conf.deleted
> Restoring client configuration files
> nscd daemon is not install
Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
More info
Krb5 log is showing:
Jun 21 20:42:47 Firstmaster.domain.local krb5kdc[2141](info): AS_REQ (4
etypes {18 17 16 23}) 10.x.x.x: LOOKING_UP_CLIENT: admin@domain.LOCAL for
krbtgt/DOMAIN.LOCAL@DOMAIN.LOCAL, Server error
[bob@Firstmaster etc]# kinit -v admin
kinit: Credentials cache file '/tmp/krb5cc_0' not found while validating
credentials
Sean Hogan
From: Sean Hogan/Durham/IBM
To: freeipa-users
Date: 06/21/2016 12:02 PM
Subject: Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
Has anyone seen these before?
First Master IPA DNS logs show: Looks like the host names are getting the
domain twice domain.local.domain.local
client 10.x.x.x#58094: query failed (SERVFAIL) for
server1.domain.local.domain.local/IN/ at query.c:6569
timeout in ldap_pool_getconnection(): try to raise 'connections' parameter;
potential deadlock?
client 10.x.x.x#44147: query failed (SERVFAIL) for
x.x.x.10.in-addr.arpa/IN/PTR at query.c:6569
timeout in ldap_pool_getconnection(): try to raise 'connections' parameter;
potential deadlock?
client 10.x.x.x#56466: query failed (SERVFAIL) for
x.x.x.10.in-addr.arpa/IN/PTR at query.c:6569
timeout in ldap_pool_getconnection(): try to raise 'connections' parameter;
potential deadlock?
client 10.x.x.x53367: query failed (SERVFAIL) for
server2.domain.local.domain.local/IN/A at query.c:6569
timeout in ldap_pool_getconnection(): try to raise 'connections' parameter;
potential deadlock?
client 10.x.x.x#53367: query failed (SERVFAIL) for
server2.domain.local.domain.local/IN/ at query.c:6569
So enrolls are failing at this point when tyring to enroll to a replica:
[bob@server1 log]# ipa-client-install –enable-dns-updates
Discovery was successful!
Hostname: server1.watson.local
Realm: DOMAIN.LOCAL
DNS Domain: domain.local
IPA Server: ipareplica.domain.local
BaseDN: dc=domain,dc=local
Continue to configure the system with these values? [no]: yes
User authorized to enroll computers: bob
Synchronizing time with KDC...
Password for bob@DOMAIN.LOCAL:
Successfully retrieved CA cert
Subject: CN=Certificate Authority,O=DOMAIN.LOCAL
Issuer: CN=Certificate Authority,O=DOMAIN.LOCAL
Valid From: Tue Jan 06 19:37:09 2015 UTC
Valid Until: Sat Jan 06 19:37:09 2035 UTC
Enrolled in IPA realm DOMAIN.LOCAL
Attempting to get host TGT...
Created /etc/ipa/default.conf
New SSSD config will be created
Configured sudoers in /etc/nsswitch.conf
Configured /etc/sssd/sssd.conf
Configured /etc/krb5.conf for IPA realm DOMAIN.LOCAL
trying https://ipareplica.domain.local/ipa/xml
Cannot connect to the server due to Kerberos error: Kerberos error:
Kerberos error: ('Unspecified GSS failure. Minor code may provide more
information', 851968)/('KDC returned error string: PROCESS_TGS',
-1765328324)/. Trying with delegate=True
trying https://ipareplica.domain.local/ipa/xml
Second connect with delegate=True also failed: Kerberos error: Kerberos
error: ('Unspecified GSS failure. Minor code may provide more
information', 851968)/('KDC returned error string: PROCESS_TGS',
-1765328324)/
Cannot connect to the IPA server XML-RPC interface: Kerberos error:
Kerberos error: ('Unspecified GSS failure. Minor code may provide more
information', 851968)/('KDC returned error string: PROCESS_TGS',
-1765328324)/
Installation failed. Rolling back changes.
Unenrolling client from IPA server
Unenrolling host failed: Error obtaining initial credentials: Generic error
(see e-text).
Removing Kerberos service principals from /etc/krb5.keytab
Disabling client Kerberos and LDAP configurations
Redundant SSSD configuration file /etc/sssd/sssd.conf was moved
to /etc/sssd/sssd.conf.deleted
Restoring client configuration files
nscd daemon is not installed, skip configuration
nslcd daemon is not installed, skip configuration
Client uninstall complete.
Sean Hogan
From: Sean Hogan/Durham/IBM
To: Sean Hogan/Durham/IBM@IBMUS
Cc: freeipa-users
Date: 06/20/2016 12:49 PM
Subject:Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
Also seeing this in the upgrade log on the first master but not on the 7
ipas.
ERROR Failed to restart named: Command '/sbin/service named restart '
returned non-zero exit status 7
which led me to
https://bugzilla.redhat.com/show_bug.cgi?id=895298
Sean Hogan
From: Sean Hogan/Durham/IBM@IBMUS
To: freeipa-users
Date: 06/20/2016 11:46 AM
Subject:Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
Sent by:freeipa-users-boun...@redhat.com
Hi All..
I thought we fixed this issue by rebooting the KVM host but it is showing
again. Our First Master IPA is being rebooted 2 -5 times a day now just to
keep it alive.
What we are seeing:
God@FirstMaster log]# kinit admin
kinit: Cannot contact any KDC for realm 'Domain.LOCAL' while getting
initial credentials
Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
Has anyone seen these before?
First Master IPA DNS logs show: Looks like the host names are getting the
domain twice domain.local.domain.local
client 10.x.x.x#58094: query failed (SERVFAIL) for
server1.domain.local.domain.local/IN/ at query.c:6569
timeout in ldap_pool_getconnection(): try to raise 'connections' parameter;
potential deadlock?
client 10.x.x.x#44147: query failed (SERVFAIL) for
x.x.x.10.in-addr.arpa/IN/PTR at query.c:6569
timeout in ldap_pool_getconnection(): try to raise 'connections' parameter;
potential deadlock?
client 10.x.x.x#56466: query failed (SERVFAIL) for
x.x.x.10.in-addr.arpa/IN/PTR at query.c:6569
timeout in ldap_pool_getconnection(): try to raise 'connections' parameter;
potential deadlock?
client 10.x.x.x53367: query failed (SERVFAIL) for
server2.domain.local.domain.local/IN/A at query.c:6569
timeout in ldap_pool_getconnection(): try to raise 'connections' parameter;
potential deadlock?
client 10.x.x.x#53367: query failed (SERVFAIL) for
server2.domain.local.domain.local/IN/ at query.c:6569
So enrolls are failing at this point when tyring to enroll to a replica:
[bob@server1 log]# ipa-client-install –enable-dns-updates
Discovery was successful!
Hostname: server1.watson.local
Realm: DOMAIN.LOCAL
DNS Domain: domain.local
IPA Server: ipareplica.domain.local
BaseDN: dc=domain,dc=local
Continue to configure the system with these values? [no]: yes
User authorized to enroll computers: bob
Synchronizing time with KDC...
Password for bob@DOMAIN.LOCAL:
Successfully retrieved CA cert
Subject: CN=Certificate Authority,O=DOMAIN.LOCAL
Issuer: CN=Certificate Authority,O=DOMAIN.LOCAL
Valid From: Tue Jan 06 19:37:09 2015 UTC
Valid Until: Sat Jan 06 19:37:09 2035 UTC
Enrolled in IPA realm DOMAIN.LOCAL
Attempting to get host TGT...
Created /etc/ipa/default.conf
New SSSD config will be created
Configured sudoers in /etc/nsswitch.conf
Configured /etc/sssd/sssd.conf
Configured /etc/krb5.conf for IPA realm DOMAIN.LOCAL
trying https://ipareplica.domain.local/ipa/xml
Cannot connect to the server due to Kerberos error: Kerberos error:
Kerberos error: ('Unspecified GSS failure. Minor code may provide more
information', 851968)/('KDC returned error string: PROCESS_TGS',
-1765328324)/. Trying with delegate=True
trying https://ipareplica.domain.local/ipa/xml
Second connect with delegate=True also failed: Kerberos error: Kerberos
error: ('Unspecified GSS failure. Minor code may provide more
information', 851968)/('KDC returned error string: PROCESS_TGS',
-1765328324)/
Cannot connect to the IPA server XML-RPC interface: Kerberos error:
Kerberos error: ('Unspecified GSS failure. Minor code may provide more
information', 851968)/('KDC returned error string: PROCESS_TGS',
-1765328324)/
Installation failed. Rolling back changes.
Unenrolling client from IPA server
Unenrolling host failed: Error obtaining initial credentials: Generic error
(see e-text).
Removing Kerberos service principals from /etc/krb5.keytab
Disabling client Kerberos and LDAP configurations
Redundant SSSD configuration file /etc/sssd/sssd.conf was moved
to /etc/sssd/sssd.conf.deleted
Restoring client configuration files
nscd daemon is not installed, skip configuration
nslcd daemon is not installed, skip configuration
Client uninstall complete.
Sean Hogan
From: Sean Hogan/Durham/IBM
To: Sean Hogan/Durham/IBM@IBMUS
Cc: freeipa-users
Date: 06/20/2016 12:49 PM
Subject:Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
Also seeing this in the upgrade log on the first master but not on the 7
ipas.
ERROR Failed to restart named: Command '/sbin/service named restart '
returned non-zero exit status 7
which led me to
https://bugzilla.redhat.com/show_bug.cgi?id=895298
Sean Hogan
From: Sean Hogan/Durham/IBM@IBMUS
To: freeipa-users
Date: 06/20/2016 11:46 AM
Subject:Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
Sent by:freeipa-users-boun...@redhat.com
Hi All..
I thought we fixed this issue by rebooting the KVM host but it is showing
again. Our First Master IPA is being rebooted 2 -5 times a day now just to
keep it alive.
What we are seeing:
God@FirstMaster log]# kinit admin
kinit: Cannot contact any KDC for realm 'Domain.LOCAL' while getting
initial credentials
DNS is not working as nslookup is failing to a replica think once we
lose DNS it all goes down hill which makes sense.
[god@FirstMaster log]# ipactl stop -> Just hangs forever.. no replies..
no error.. nothing
I try service named stop and nothing happens
I have the box hard shutdown from KVM console. Reboot it and it works for a
little while but eventually back to same behavior.
At this point I can service named stop and it responds... ipactl status and
it responds.. but when if I try service named restart I get
[god@Firs
Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
Noticed something else really goofy in the DNS logs on master ipa
client 10.9.0.1#58094: query failed (SERVFAIL) for
serv1.domain.local.domain.local/IN/ at query.c:6569
timeout in ldap_pool_getconnection(): try to raise 'connections' parameter;
potential deadlock?
client 10.10.0.1#44147: query failed (SERVFAIL) for
serv2.in-addr.arpa/IN/PTR at query.c:6569
timeout in ldap_pool_getconnection(): try to raise 'connections' parameter;
potential deadlock?
client 10.10.0.1#56466: query failed (SERVFAIL) for
serv2.in-addr.arpa/IN/PTR at query.c:6569
timeout in ldap_pool_getconnection(): try to raise 'connections' parameter;
potential deadlock?
client 10.110.0.1#53367: query failed (SERVFAIL) for
serv3.domain.local.domain.local/IN/A at query.c:6569
timeout in ldap_pool_getconnection(): try to raise 'connections' parameter;
potential deadlock?
client 10.110.0.1#53367: query failed (SERVFAIL) for
serv3.domain.local.domain.local/IN/ at query.c:6569
On a replica I see this
[bob@replica2 data]# tail -f named.run
dispatch 0x7f408c187970: open_socket(0.0.0.0#1935) -> permission denied:
continuing
dispatch 0x7f408c187970: open_socket(0.0.0.0#8610) -> permission denied:
continuing
dispatch 0x7f408c187970: open_socket(0.0.0.0#6514) -> permission denied:
continuing
dispatch 0x7f408c187970: open_socket(0.0.0.0#8610) -> permission denied:
continuing
dispatch 0x7f408c187970: open_socket(0.0.0.0#1935) -> permission denied:
continuing
Feel like I am caught in a troubleshooting loop being to close to it.. has
anyone seen this before?
Sean Hogan
From: Sean Hogan/Durham/IBM
To: Sean Hogan/Durham/IBM@IBMUS
Cc: freeipa-users
Date: 06/20/2016 12:49 PM
Subject: Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
Also seeing this in the upgrade log on the first master but not on the 7
ipas.
ERROR Failed to restart named: Command '/sbin/service named restart '
returned non-zero exit status 7
which led me to
https://bugzilla.redhat.com/show_bug.cgi?id=895298
Sean Hogan
From: Sean Hogan/Durham/IBM@IBMUS
To: freeipa-users
Date: 06/20/2016 11:46 AM
Subject:Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
Sent by:freeipa-users-boun...@redhat.com
Hi All..
I thought we fixed this issue by rebooting the KVM host but it is showing
again. Our First Master IPA is being rebooted 2 -5 times a day now just to
keep it alive.
What we are seeing:
God@FirstMaster log]# kinit admin
kinit: Cannot contact any KDC for realm 'Domain.LOCAL' while getting
initial credentials
DNS is not working as nslookup is failing to a replica think once we
lose DNS it all goes down hill which makes sense.
[god@FirstMaster log]# ipactl stop -> Just hangs forever.. no replies..
no error.. nothing
I try service named stop and nothing happens
I have the box hard shutdown from KVM console. Reboot it and it works for a
little while but eventually back to same behavior.
At this point I can service named stop and it responds... ipactl status and
it responds.. but when if I try service named restart I get
[god@FirstMaster log]# service named stop
Stopping named: ..
[god@Firstmaster log]# service named start
Starting named: [FAILED]
[god@FirstMaster log]# service named status
rndc: connect failed: 127.0.0.1#953: connection refused
named dead but pid file exists
Rebooted box and it is hung on shutting down domain-local and never fully
shuts down.. have to get it hard shutdown again.
During an attempt to gracefully shut down we see this
Shutting Down dirsrv:
PKI-IPA OK
DOMAIN-LOCAL FAILED
*** Error: 1 instance(s) unsuccessfully stopped FAILED
Then it moves on to shut other things down and returns to dirsrv
Shutting Down dirsrv:
PKI-IPAserver already stopped FAILED {Makes sense.. it died earlier}
DOMAIN-LOCAL... {this sits here til we hard shutdown}
bind-libs-9.8.2-0.47.rc1.el6.x86_64
bind-9.8.2-0.47.rc1.el6.x86_64
bind-utils-9.8.2-0.47.rc1.el6.x86_64
ipa-client-3.0.0-50.el6.1.x86_64
ipa-server-selinux-3.0.0-50.el6.1.x86_64
ipa-server-3.0.0-50.el6.1.x86_64
sssd-ipa-1.13.3-22.el6.x86_64
/var/log/dirsrv/slapd-DOMAIN-LOCAL
[20/Jun/2016:13:29:06 -0400] - 389-Directory/1.2.11.15 B2016.063.2110
starting up
[20/Jun/2016:13:29:06 -0400] schema-compat-plugin - warning: no entries set
up under cn=computers, cn=compat,dc=domain,dc=local
[20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin - ruv_compare_ruv: RUV
[database RUV] does not contain element [{replica 7} 55ca26a90007
5688d8e60017] which is present in RUV [changelog max RUV]
[20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin -
replica_check_for_data_reload: Warning: for replica dc=domain,dc=local
there were some differences between the changelog max RUV and the database
RUV. If there are obsolete elements in the database RUV, you should remove
them using the CLEANALLRUV task. If they are not obsolete, you shoul
Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
Also seeing this in the upgrade log on the first master but not on the 7
ipas.
ERROR Failed to restart named: Command '/sbin/service named restart '
returned non-zero exit status 7
which led me to
https://bugzilla.redhat.com/show_bug.cgi?id=895298
Sean Hogan
From: Sean Hogan/Durham/IBM@IBMUS
To: freeipa-users
Date: 06/20/2016 11:46 AM
Subject: Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
Sent by:freeipa-users-boun...@redhat.com
Hi All..
I thought we fixed this issue by rebooting the KVM host but it is showing
again. Our First Master IPA is being rebooted 2 -5 times a day now just to
keep it alive.
What we are seeing:
God@FirstMaster log]# kinit admin
kinit: Cannot contact any KDC for realm 'Domain.LOCAL' while getting
initial credentials
DNS is not working as nslookup is failing to a replica think once we
lose DNS it all goes down hill which makes sense.
[god@FirstMaster log]# ipactl stop -> Just hangs forever.. no replies..
no error.. nothing
I try service named stop and nothing happens
I have the box hard shutdown from KVM console. Reboot it and it works for a
little while but eventually back to same behavior.
At this point I can service named stop and it responds... ipactl status and
it responds.. but when if I try service named restart I get
[god@FirstMaster log]# service named stop
Stopping named: ..
[god@Firstmaster log]# service named start
Starting named: [FAILED]
[god@FirstMaster log]# service named status
rndc: connect failed: 127.0.0.1#953: connection refused
named dead but pid file exists
Rebooted box and it is hung on shutting down domain-local and never fully
shuts down.. have to get it hard shutdown again.
During an attempt to gracefully shut down we see this
Shutting Down dirsrv:
PKI-IPA OK
DOMAIN-LOCAL FAILED
*** Error: 1 instance(s) unsuccessfully stopped FAILED
Then it moves on to shut other things down and returns to dirsrv
Shutting Down dirsrv:
PKI-IPAserver already stopped FAILED {Makes sense.. it died earlier}
DOMAIN-LOCAL... {this sits here til we hard shutdown}
bind-libs-9.8.2-0.47.rc1.el6.x86_64
bind-9.8.2-0.47.rc1.el6.x86_64
bind-utils-9.8.2-0.47.rc1.el6.x86_64
ipa-client-3.0.0-50.el6.1.x86_64
ipa-server-selinux-3.0.0-50.el6.1.x86_64
ipa-server-3.0.0-50.el6.1.x86_64
sssd-ipa-1.13.3-22.el6.x86_64
/var/log/dirsrv/slapd-DOMAIN-LOCAL
[20/Jun/2016:13:29:06 -0400] - 389-Directory/1.2.11.15 B2016.063.2110
starting up
[20/Jun/2016:13:29:06 -0400] schema-compat-plugin - warning: no entries set
up under cn=computers, cn=compat,dc=domain,dc=local
[20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin - ruv_compare_ruv: RUV
[database RUV] does not contain element [{replica 7} 55ca26a90007
5688d8e60017] which is present in RUV [changelog max RUV]
[20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin -
replica_check_for_data_reload: Warning: for replica dc=domain,dc=local
there were some differences between the changelog max RUV and the database
RUV. If there are obsolete elements in the database RUV, you should remove
them using the CLEANALLRUV task. If they are not obsolete, you should check
their status to see why there are no changes from those servers in the
changelog.
[20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial
credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in
keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
for requested realm)
[20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial
credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in
keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
for requested realm)
[20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial
credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in
keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
for requested realm)
[20/Jun/2016:13:29:07 -0400] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2
(Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
failure. Minor code may provide more information (Credentials cache file
'/tmp/krb5cc_495' not found)) errno 2 (No such file or directory)
[20/Jun/2016:13:29:07 -0400] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
[20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin -
agmt="cn=meToserver8.domain.local" (server8:389): Replication bind with
GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure:
GSSAPI Error: Unspecified GSS failure. Minor code may provide more
information (Credentials cache file '/tmp/krb5cc_495' not found))
[20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial
credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in
keytab [FILE:/etc/dirsrv/ds.keytab]: -1
Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
Hi All..
I thought we fixed this issue by rebooting the KVM host but it is showing
again. Our First Master IPA is being rebooted 2 -5 times a day now just to
keep it alive.
What we are seeing:
God@FirstMaster log]# kinit admin
kinit: Cannot contact any KDC for realm 'Domain.LOCAL' while getting
initial credentials
DNS is not working as nslookup is failing to a replica think once we
lose DNS it all goes down hill which makes sense.
[god@FirstMaster log]# ipactl stop -> Just hangs forever.. no
replies.. no error.. nothing
I try service named stop and nothing happens
I have the box hard shutdown from KVM console. Reboot it and it works for
a little while but eventually back to same behavior.
At this point I can service named stop and it responds... ipactl status and
it responds.. but when if I try service named restart I get
[god@FirstMaster log]# service named stop
Stopping named: ..
[god@Firstmaster log]# service named start
Starting named:[FAILED]
[god@FirstMaster log]# service named status
rndc: connect failed: 127.0.0.1#953: connection refused
named dead but pid file exists
Rebooted box and it is hung on shutting down domain-local and never fully
shuts down.. have to get it hard shutdown again.
During an attempt to gracefully shut down we see this
Shutting Down dirsrv:
PKI-IPA OK
DOMAIN-LOCALFAILED
*** Error: 1 instance(s) unsuccessfully stopped FAILED
Then it moves on to shut other things down and returns to dirsrv
Shutting Down dirsrv:
PKI-IPAserver already stopped
FAILED {Makes sense.. it died earlier}
DOMAIN-LOCAL...
{this sits here til we hard shutdown}
bind-libs-9.8.2-0.47.rc1.el6.x86_64
bind-9.8.2-0.47.rc1.el6.x86_64
bind-utils-9.8.2-0.47.rc1.el6.x86_64
ipa-client-3.0.0-50.el6.1.x86_64
ipa-server-selinux-3.0.0-50.el6.1.x86_64
ipa-server-3.0.0-50.el6.1.x86_64
sssd-ipa-1.13.3-22.el6.x86_64
/var/log/dirsrv/slapd-DOMAIN-LOCAL
[20/Jun/2016:13:29:06 -0400] - 389-Directory/1.2.11.15 B2016.063.2110
starting up
[20/Jun/2016:13:29:06 -0400] schema-compat-plugin - warning: no entries set
up under cn=computers, cn=compat,dc=domain,dc=local
[20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin - ruv_compare_ruv: RUV
[database RUV] does not contain element [{replica 7} 55ca26a90007
5688d8e60017] which is present in RUV [changelog max RUV]
[20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin -
replica_check_for_data_reload: Warning: for replica dc=domain,dc=local
there were some differences between the changelog max RUV and the database
RUV. If there are obsolete elements in the database RUV, you should remove
them using the CLEANALLRUV task. If they are not obsolete, you should
check their status to see why there are no changes from those servers in
the changelog.
[20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial
credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in
keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
for requested realm)
[20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial
credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in
keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
for requested realm)
[20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial
credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in
keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
for requested realm)
[20/Jun/2016:13:29:07 -0400] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2
(Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
failure. Minor code may provide more information (Credentials cache file
'/tmp/krb5cc_495' not found)) errno 2 (No such file or directory)
[20/Jun/2016:13:29:07 -0400] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
[20/Jun/2016:13:29:07 -0400] NSMMReplicationPlugin -
agmt="cn=meToserver8.domain.local" (server8:389): Replication bind with
GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure:
GSSAPI Error: Unspecified GSS failure. Minor code may provide more
information (Credentials cache file '/tmp/krb5cc_495' not found))
[20/Jun/2016:13:29:07 -0400] set_krb5_creds - Could not get initial
credentials for principal [ldap/server1.domain.local@DOMAIN.LOCAL] in
keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
for requested realm)
[20/Jun/2016:13:29:07 -0400] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2
(Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
failure. Minor code may provide more information (Credentials cache file
'/tmp/krb5cc_495' not found)) errno 2 (No such file or directory)
[20
Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
Hi Robert.. Thanks for the reply. Think I might have found the issue. The KVM host my master was running on was showing redhat release 6.5 but the libvrt packages were showing 6.6. I think the managers of the kvm host did not reboot it after an update with new kernel. Asked them to reboot the KVM host after I gracefully shut down my NFS profile server and Master IPA (both run on that host). However Master IPA would not shutdown so they rebooted it with the IPA server still running. Once it was back up and the 2 servers were back up I had to gracefully shutdown the Master IPA and this time it did shutdown. Powered back up and it seems to be running fine now. BTW... there is a lot of info in the upgrade log but will overview it more later. Thanks Sean Hogan From: Rob Crittenden To: Sean Hogan/Durham/IBM@IBMUS, freeipa-users Date: 06/02/2016 02:42 PM Subject:Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem Sean Hogan wrote: > Hello All, > > Recently went from RHEL 6.7 IPA 3.0.47 to 6.8 IPA 3.0.50. I also think > (not sure on this yet) that they changed ntp.. ntp used to point at my > ipas.. but they look like they are now pointing elsewhere. Everything > was stable at 6.7 3.0.47 pointing to IPA for NTP. However.. they all > seem to have the same date. > > > My master first IPA is acting up. Replication is off, kerberos seems to > be off, DNS is off and I think IPA in general on it is toast. > We do have 8 IPAs.. only FirstMaster is acting up it seems right now and > all either running on KVM or ESXI. > > > [God@FirstMasterIPA slapd-DOMAIN-LOCAL]# kinit admin > kinit: Generic error (see e-text) while getting initial credential ipactl status should show what services are running. It looks like the KDC is responding but can't talk to the LDAP backend. > > > slapd-DOMAIN-LOCAL > [01/Jun/2016:18:25:43 -0400] slapd_ldap_sasl_interactive_bind - Error: > could not perform interactive bind for id [] mech [GSSAPI]: LDAP error > -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified > GSS failure. Minor code may provide more information (Cannot contact any > KDC for realm 'DOMAIN.LOCAL')) errno 115 (Operation now in progress) > [01/Jun/2016:18:25:43 -0400] slapi_ldap_bind - Error: could not perform > interactive bind for id [] mech [GSSAPI]: error -2 (Local error) > [01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin - > agmt="cn=meToipaserv2.domain.local" (ipaserv2:389): Replication bind > with GSSAPI auth resumed > [01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin - > agmt="cn=meToipaserv3.domain.local" (ipaserv3:389): Replication bind > with GSSAPI auth resumed > [01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin - > agmt="cn=meToipaserv4.domain.local" (ipaserv4:389): Replication bind > with GSSAPI auth resumed > [01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin - > agmt="cn=meToipaserv5.domain.local" (ipaserv5:389): Replication bind > with GSSAPI auth resumed > [01/Jun/2016:18:28:04 -0400] slapd_ldap_sasl_interactive_bind - Error: > could not perform interactive bind for id [] mech [GSSAPI]: LDAP error > 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI > Failure: gss_accept_sec_context) errno 0 (Success) > [01/Jun/2016:18:28:04 -0400] slapi_ldap_bind - Error: could not perform > interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials) > [01/Jun/2016:18:28:13 -0400] slapd_ldap_sasl_interactive_bind - Error: > could not perform interactive bind for id [] mech [GSSAPI]: LDAP error > -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified > GSS failure. Minor code may provide more information (No credentials > cache found)) errno 2 (No such file or directory) > [01/Jun/2016:18:28:13 -0400] slapi_ldap_bind - Error: could not perform > interactive bind for id [] mech [GSSAPI]: error -2 (Local error) > [01/Jun/2016:18:33:03 -0400] slapd_ldap_sasl_interactive_bind - Error: > could not perform interactive bind for id [] mech [GSSAPI]: LDAP error > 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI > Failure: gss_accept_sec_context) errno 0 (Success) > [01/Jun/2016:18:33:03 -0400] slapi_ldap_bind - Error: could not perform > interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials) > [01/Jun/2016:18:33:18 -0400] slapd_ldap_sasl_interactive_bind - Error: > could not perform interactive bind for id [] mech [GSSAPI]: LDAP error > -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified > GSS failure. Minor code may provide more information (No credentials > cache found)) errno 2 (No such file or directory) > [01/Jun/2016:18:33:18 -0400] slapi_ldap_bind - Error: could not perform > interactive bind for i
Re: [Freeipa-users] IPA 3.0.47 to 3.0.50 Upgrade problem
Sean Hogan wrote: Hello All, Recently went from RHEL 6.7 IPA 3.0.47 to 6.8 IPA 3.0.50. I also think (not sure on this yet) that they changed ntp.. ntp used to point at my ipas.. but they look like they are now pointing elsewhere. Everything was stable at 6.7 3.0.47 pointing to IPA for NTP. However.. they all seem to have the same date. My master first IPA is acting up. Replication is off, kerberos seems to be off, DNS is off and I think IPA in general on it is toast. We do have 8 IPAs.. only FirstMaster is acting up it seems right now and all either running on KVM or ESXI. [God@FirstMasterIPA slapd-DOMAIN-LOCAL]# kinit admin kinit: Generic error (see e-text) while getting initial credential ipactl status should show what services are running. It looks like the KDC is responding but can't talk to the LDAP backend. slapd-DOMAIN-LOCAL [01/Jun/2016:18:25:43 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Cannot contact any KDC for realm 'DOMAIN.LOCAL')) errno 115 (Operation now in progress) [01/Jun/2016:18:25:43 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin - agmt="cn=meToipaserv2.domain.local" (ipaserv2:389): Replication bind with GSSAPI auth resumed [01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin - agmt="cn=meToipaserv3.domain.local" (ipaserv3:389): Replication bind with GSSAPI auth resumed [01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin - agmt="cn=meToipaserv4.domain.local" (ipaserv4:389): Replication bind with GSSAPI auth resumed [01/Jun/2016:18:25:48 -0400] NSMMReplicationPlugin - agmt="cn=meToipaserv5.domain.local" (ipaserv5:389): Replication bind with GSSAPI auth resumed [01/Jun/2016:18:28:04 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context) errno 0 (Success) [01/Jun/2016:18:28:04 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials) [01/Jun/2016:18:28:13 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No credentials cache found)) errno 2 (No such file or directory) [01/Jun/2016:18:28:13 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [01/Jun/2016:18:33:03 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context) errno 0 (Success) [01/Jun/2016:18:33:03 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error 49 (Invalid credentials) [01/Jun/2016:18:33:18 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No credentials cache found)) errno 2 (No such file or directory) [01/Jun/2016:18:33:18 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [01/Jun/2016:18:38:03 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context) errno 0 (Success) And this makes it look like it can't talk to the KDC. I'd check for SELinux errors, ausearch -m AVC -ts recent I think the rest is just indication that something is wrong with either the LDAP servers, the KDC or both. You may also want to look at /var/log/ipaupgrade.log to ensure that the upgrade was successful. rob [God@FirstMasterIPA slapd-DOMAIN-LOCAL]# ipa-replica-manage -v list --> just hangs and never returns [God@FirstMasterIPA slapd-DOMAIN-LOCAL]# ipactl start ->Just hangs here as well.. never gets to the KDC. Starting Directory Service Starting dirsrv: PKI-IPA... already running [ OK ] DOMAIN-LOCAL... already running [ OK ] If I run nslookup it fails over to a Replica for the DNS resolution instead of resolving ips itself. PKI log shows a bunch of this: [02/Jun/2016:11:15:25 -0400] NSMMReplicationPlugin - agmt="cn=masterAgreement1-ipaserver2.domain.local-pki-ca" (ipaserver2:7389): Replication bind with SIMPLE auth failed: LDAP error -1 (Can't contact LDAP server) ((null)) [02/Jun/2
