[Freeipa-users] Incremental update failed and requires administrator action

[bahan w]

Hello !

I recently installed a replica (master2) in addition of my master (master1)
with IPA 3.0.0-47 on RHEL6.6.
I don't know from when exactly, but the dirsrv (and the whole ipa service)
on master1 crashes regularly with the following logs.

###
[22/Jan/2016:15:38:20 +0100] - 389-Directory/1.2.11.15 B2015.279.183
starting up
[22/Jan/2016:15:38:20 +0100] schema-compat-plugin - warning: no entries set
up under cn=computers, cn=compat,dc=
[22/Jan/2016:15:38:21 +0100] schema-compat-plugin - warning: no entries set
up under cn=ng, cn=compat,dc=
[22/Jan/2016:15:38:21 +0100] schema-compat-plugin - warning: no entries set
up under ou=sudoers,dc=
[22/Jan/2016:15:38:21 +0100] - slapd started.  Listening on All Interfaces
port 389 for LDAP requests
[22/Jan/2016:15:38:21 +0100] - Listening on All Interfaces port 636 for
LDAPS requests
[22/Jan/2016:15:38:21 +0100] - Listening on /var/run/slapd-.socket
for LDAPI requests
[22/Jan/2016:17:04:03 +0100] NSMMReplicationPlugin - changelog program -
_cl5WriteOperationTxn: retry (49) the transaction
(csn=56a252ef0004) failed (rc=-30994 (DB_LOCK_DEADLOCK: Locker
killed to resolve a deadlock))
[22/Jan/2016:17:04:03 +0100] NSMMReplicationPlugin - changelog program -
_cl5WriteOperationTxn: failed to write entry with csn
(56a252ef0004); db error - -30994 DB_LOCK_DEADLOCK: Locker killed
to resolve a deadlock
[22/Jan/2016:17:04:03 +0100] NSMMReplicationPlugin -
write_changelog_and_ruv: can't add a change for
uid=,cn=users,cn=accounts,dc= (uniqid:
a7ebd403-c12111e5-9c84c092-9a5deb81, optype: 16) to changelog csn
56a252ef0004
[22/Jan/2016:17:04:03 +0100] NSMMReplicationPlugin -
agmt="cn=meTo" (:389): Missing data encountered
[22/Jan/2016:17:04:03 +0100] NSMMReplicationPlugin -
agmt="cn=meTo" (:389): Incremental update
failed and requires administrator action
###

Then the dirsrv, I mean the whole ipa server, is down.
When I restart the service, here is what is see :

###
[22/Jan/2016:17:06:18 +0100] - 389-Directory/1.2.11.15 B2015.279.183
starting up
[22/Jan/2016:17:06:18 +0100] - Detected Disorderly Shutdown last time
Directory Server was running, recovering database.
[22/Jan/2016:17:06:18 +0100] schema-compat-plugin - warning: no entries set
up under cn=computers, cn=compat,dc=
[22/Jan/2016:17:06:19 +0100] schema-compat-plugin - warning: no entries set
up under cn=ng, cn=compat,dc=
[22/Jan/2016:17:06:19 +0100] schema-compat-plugin - warning: no entries set
up under ou=sudoers,dc=
[22/Jan/2016:17:06:20 +0100] set_krb5_creds - Could not get initial
credentials for principal [ldap/@] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328324 (Generic error (see e-text))
[22/Jan/2016:17:06:20 +0100] - slapd started.  Listening on All Interfaces
port 389 for LDAP requests
[22/Jan/2016:17:06:20 +0100] - Listening on All Interfaces port 636 for
LDAPS requests
[22/Jan/2016:17:06:20 +0100] - Listening on /var/run/slapd-.socket
for LDAPI requests
[22/Jan/2016:17:06:20 +0100] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2
(Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
failure.  Minor code may provide more information (Credentials cache file
'/tmp/krb5cc_244' not found)) errno 0 (Success)
[22/Jan/2016:17:06:20 +0100] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
[22/Jan/2016:17:06:20 +0100] NSMMReplicationPlugin -
agmt="cn=meTo" (:389): Replication bind with
GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure:
GSSAPI Error: Unspecified GSS failure.  Minor code may provide more
information (Credentials cache file '/tmp/krb5cc_244' not found))
[22/Jan/2016:17:06:23 +0100] NSMMReplicationPlugin -
agmt="cn=meTo" (:389): Replication bind with
GSSAPI auth resumed
###

It seems that there is a problem to write an entry in the DB ? Do you know
how I can solve this problem please ?

Furthermore, it seems that there is a second problem with the keytab
/etc/dirsrv/ds.keytab.

The keytab is good for me :
###
#ls -l /etc/dirsrv/ds.keytab
-rw--- 1 dirsrv dirsrv 362 Jan 21 14:12 /etc/dirsrv/ds.keytab
# kinit -kt /etc/dirsrv/ds.keytab ldap/@
# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: ldap/@

Valid starting ExpiresService principal
01/25/16 11:54:23  01/26/16 11:54:23  krbtgt/@
###

I wonder if this second problem does not come from the user dirsrv who
would not be able to use this keytab.
I cannot test this because this user dirsrv has been created with nologin.
###
# su - dirsrv -c "kinit -kt /etc/dirsrv/ds.keytab ldap/@"
This account is currently not available.

# grep dirsrv /etc/passwd
dirsrv:x:244:497::/var/lib/dirsrv:/sbin/nologin
pkisrv:x:246:497::/var/lib/dirsrv:/sbin/nologin
###

Just for my information, is it normal that these users are created with
nologin ?

Best regards.

Bahan
-- 
Manage your subscription for the Freeipa-users mailing list:

Re: [Freeipa-users] Incremental update failed and requires administrator action

[Ludwig Krispenz]

could you get a core dump from the crash:
http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes

Ludwig

On 01/25/2016 12:08 PM, bahan w wrote:

Hello !

I recently installed a replica (master2) in addition of my master 
(master1) with IPA 3.0.0-47 on RHEL6.6.
I don't know from when exactly, but the dirsrv (and the whole ipa 
service) on master1 crashes regularly with the following logs.


###
[22/Jan/2016:15:38:20 +0100] - 389-Directory/1.2.11.15 
 B2015.279.183 starting up
[22/Jan/2016:15:38:20 +0100] schema-compat-plugin - warning: no 
entries set up under cn=computers, cn=compat,dc=
[22/Jan/2016:15:38:21 +0100] schema-compat-plugin - warning: no 
entries set up under cn=ng, cn=compat,dc=
[22/Jan/2016:15:38:21 +0100] schema-compat-plugin - warning: no 
entries set up under ou=sudoers,dc=
[22/Jan/2016:15:38:21 +0100] - slapd started.  Listening on All 
Interfaces port 389 for LDAP requests
[22/Jan/2016:15:38:21 +0100] - Listening on All Interfaces port 636 
for LDAPS requests
[22/Jan/2016:15:38:21 +0100] - Listening on 
/var/run/slapd-.socket for LDAPI requests
[22/Jan/2016:17:04:03 +0100] NSMMReplicationPlugin - changelog program 
- _cl5WriteOperationTxn: retry (49) the transaction 
(csn=56a252ef0004) failed (rc=-30994 (DB_LOCK_DEADLOCK: Locker 
killed to resolve a deadlock))
[22/Jan/2016:17:04:03 +0100] NSMMReplicationPlugin - changelog program 
- _cl5WriteOperationTxn: failed to write entry with csn 
(56a252ef0004); db error - -30994 DB_LOCK_DEADLOCK: Locker 
killed to resolve a deadlock
[22/Jan/2016:17:04:03 +0100] NSMMReplicationPlugin - 
write_changelog_and_ruv: can't add a change for 
uid=,cn=users,cn=accounts,dc= (uniqid: 
a7ebd403-c12111e5-9c84c092-9a5deb81, optype: 16) to changelog csn 
56a252ef0004
[22/Jan/2016:17:04:03 +0100] NSMMReplicationPlugin - 
agmt="cn=meTo" (:389): Missing data 
encountered
[22/Jan/2016:17:04:03 +0100] NSMMReplicationPlugin - 
agmt="cn=meTo" (:389): Incremental update 
failed and requires administrator action

###

Then the dirsrv, I mean the whole ipa server, is down.
When I restart the service, here is what is see :

###
[22/Jan/2016:17:06:18 +0100] - 389-Directory/1.2.11.15 
 B2015.279.183 starting up
[22/Jan/2016:17:06:18 +0100] - Detected Disorderly Shutdown last time 
Directory Server was running, recovering database.
[22/Jan/2016:17:06:18 +0100] schema-compat-plugin - warning: no 
entries set up under cn=computers, cn=compat,dc=
[22/Jan/2016:17:06:19 +0100] schema-compat-plugin - warning: no 
entries set up under cn=ng, cn=compat,dc=
[22/Jan/2016:17:06:19 +0100] schema-compat-plugin - warning: no 
entries set up under ou=sudoers,dc=
[22/Jan/2016:17:06:20 +0100] set_krb5_creds - Could not get initial 
credentials for principal [ldap/@] in keytab 
[FILE:/etc/dirsrv/ds.keytab]: -1765328324 (Generic error (see e-text))
[22/Jan/2016:17:06:20 +0100] - slapd started.  Listening on All 
Interfaces port 389 for LDAP requests
[22/Jan/2016:17:06:20 +0100] - Listening on All Interfaces port 636 
for LDAPS requests
[22/Jan/2016:17:06:20 +0100] - Listening on 
/var/run/slapd-.socket for LDAPI requests
[22/Jan/2016:17:06:20 +0100] slapd_ldap_sasl_interactive_bind - Error: 
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 
-2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified 
GSS failure.  Minor code may provide more information (Credentials 
cache file '/tmp/krb5cc_244' not found)) errno 0 (Success)
[22/Jan/2016:17:06:20 +0100] slapi_ldap_bind - Error: could not 
perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
[22/Jan/2016:17:06:20 +0100] NSMMReplicationPlugin - 
agmt="cn=meTo" (:389): Replication bind 
with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): 
generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code 
may provide more information (Credentials cache file '/tmp/krb5cc_244' 
not found))
[22/Jan/2016:17:06:23 +0100] NSMMReplicationPlugin - 
agmt="cn=meTo" (:389): Replication bind 
with GSSAPI auth resumed

###

It seems that there is a problem to write an entry in the DB ? Do you 
know how I can solve this problem please ?


Furthermore, it seems that there is a second problem with the keytab 
/etc/dirsrv/ds.keytab.


The keytab is good for me :
###
#ls -l /etc/dirsrv/ds.keytab
-rw--- 1 dirsrv dirsrv 362 Jan 21 14:12 /etc/dirsrv/ds.keytab
# kinit -kt /etc/dirsrv/ds.keytab ldap/@
# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: ldap/@

Valid starting ExpiresService principal
01/25/16 11:54:23  01/26/16 11:54:23 krbtgt/@
###

I wonder if this second problem does not come from the user dirsrv who 
would not be able to use this keytab.

I cannot test this because this user dirsrv has been created with nologin.
###
# su - dirsrv -c "kinit -kt /etc/dirsrv/ds.keytab 
ldap/@"

This account is currently not available.

# grep dirsrv /etc/passwd