Re: [Freeipa-users] Replica cannot be reinitialized after upgrade

[Ludwig Krispenz]

On 05/18/2017 10:13 PM, Goran Marik wrote:

Thanks Ludwig for the suggestion and thanks to Maciej for the confirmation from 
his end. This issue is happening for us for several weeks, so I don’t think 
this is a transient problem.

What is the best way to sanitize the logs without removing useful info before sending them 
your way? Will the files mentioned on 
"https://www.freeipa.org/page/Files_to_be_attached_to_bug_report -> Directory server 
failed" be sufficient?
yes, but we need soem additional info on the replication config and 
state, you could add /etc/dirsrv/slapd-*/dse.ldif

and the result of these query

ldapsearch -o ldif-wrap=no  -D "cn=directory 
manager" ... -b "cn=config" "objectclass=nsds5replica" \* nsds50ruv


But looking again at the csn reorted missing it is from June, 2016. So I 
wonder if this is for an stale/removed replica and cleaning the ruvs 
would help


I’ve also run the ipa_consistency_check script, and the output shows that 
something is indeed wrong with the sync:
“””
FreeIPA servers:inf01inf01inf02inf02STATE
=
Active Users15   15   15   15   OK
Stage Users 0000OK
Preserved Users 3333OK
User Groups 9999OK
Hosts   45   45   45   46   FAIL
Host Groups 7777OK
HBAC Rules  6666OK
SUDO Rules  7777OK
DNS Zones   33   33   33   33   OK
LDAP Conflicts  NO   NO   NO   NO   OK
Ghost Replicas  2222FAIL
Anonymous BIND  YES  YES  YES  YES  OK
Replication Status  inf01.prod 0inf01.dev 0inf01.dev 0inf01.dev 0
 inf02.dev 0inf02.dev 0inf01.prod 0inf01.prod 0
 inf02.prod 0inf02.prod 0inf02.prod 0inf02.dev 0
=
“””

Thanks,
Goran


On May 15, 2017, at 6:35 AM, Ludwig Krispenz <lkris...@redhat.com> wrote:

The messages you see could be transient messages, and if replication is working 
than this seems to be the case. If not we would need more data to investigate: 
deployment info, relicaIDs of all servers, ruvs, logs,.

Here is some background info: there are some scenarios where a csn could not be 
found in the changelog, eg if updates were aplied on the supplier during a 
total init, they could be part of the data and database ruv, but not in the 
changelog of the initialized replica.
ds did try to use an alternative csn in cases where it could not be found, but 
this had the risk of missing updates, so we decided to change it and make this 
misssing csn a non fatal error, backoff and retry, if another supplier would 
have updated the replica in between, the starting csn could have changed and be 
found. so if the reported missing csns change and replication continues 
everything is ok, although I think the messages should stop at some point.

There is a configuration parameter for a replciation agreement to trigger the 
previous behaviour of picking an alternative csn:
nsds5ReplicaIgnoreMissingChange
with potential values "once", "always".

where "once" just tries to kickstart replication by using another csn and 
"always" changes the default behaviour


On 05/11/2017 06:53 PM, Goran Marik wrote:

Hi,

After an upgrade to Centos 7.3.1611 with “yum update", we started seeing the 
following messages in the logs:
“””
May  9 21:58:28 inf01 ns-slapd[4323]: [09/May/2017:21:58:28.519724479 +] 
NSMMReplicationPlugin - changelog program - 
agmt="cn=cloneAgreement1-inf02.dev.ecobee.com-pki-tomcat" (inf02:389): CSN 
576b34e8000a050f not found, we aren't as up to date, or we purged
May  9 21:58:28 inf01 ns-slapd[4323]: [09/May/2017:21:58:28.550459233 +] 
NSMMReplicationPlugin - 
agmt="cn=cloneAgreement1-inf02.dev.ecobee.com-pki-tomcat" (inf02:389): Data 
required to update replica has been purged from the changelog. The replica must be 
reinitialized.
May  9 21:58:32 inf01 ns-slapd[4323]: [09/May/2017:21:58:32.588245476 +] 
agmt="cn=cloneAgreement1-inf02.dev.ecobee.com-pki-tomcat" (inf02:389) - Can't 
locate CSN 576b34e8000a050f in the changelog (DB rc=-30988). If replication stops, 
the consumer may need to be reinitialized.
May  9 21:58:32 inf01 ns-slapd[4323]: [09/May/2017:21:58:32.611400689 +] 
NSMMReplicationPlugin - changelog program - 
agmt="cn=cloneAgreement1-inf02.dev.ecobee.com-pki-tomcat" (inf02:389): CSN 
576b34e8000a050f not found, we aren't as up to date, or we purged
May  9 21:58:32 inf01 ns-slapd[4323]: [09/May/2017:21:58:32.642226385 +] 
NSM

Re: [Freeipa-users] Cant locate CSN after yum update

[Ludwig Krispenz]

On 05/18/2017 05:35 PM, Christophe TREFOIS wrote:

Dear Ludwig,

Thanks for your help in IRC to guide me in running the right commands.

Here is the output, toto1 and toto2 are CA master, and toto3 and toto4 
are non CA master. The problematic replica was toto3, and after 
re-init, we haven’t seen any errors in the log anymore.


https://paste.fedoraproject.org/paste/j8c30CZPyh8rPymjbKSvZF5M1UNdIGYhyRLivL9gydE=

I also ran ipa-replica-manage on all replicas to all replicas, so 
total of 16 command, and found all of them reported “incremental 
update succeeded”.


As discussed, I’m not sure what I’m looking at with the RUV stuff 
above, and any explanation for a newcomer to ldap / ds / freeipa would 
be greatly appreciated.

ok, here is a quick explanation of the csn/ruv stuff.

each change applied on a server gets a CSN (change sequence number), it 
basically consists of a timestamp and an identifier of the replica where 
it was originally applied, so in 59095fe1000b0012 there is a time 
stamp: 59095fe1 and a replicaid: 0012 == 18, the rest of the csn isused 
to serialize csns within the one second resolution of a timestamp.
a change is applied to the main database and written to the changelog, 
with the csn as key.


now each replica keeps track of the latest csn it has seen for each 
replicaID, so you get a vector of max csns, this is called RUV (replica 
update vector).
In a replication session, the supplier compares its own ruv with the ruv 
of the consumer and so decides if it has changes which the consumer has 
not yet seen.

based on the consumer ruv it determines the start csn to send updates.




Thanks a lot for your help!

Kind regards,
Christophe aka Trefex

On 18 May 2017, at 17:04, Christophe TREFOIS 
<christophe.tref...@uni.lu <mailto:christophe.tref...@uni.lu>> wrote:


Hi Ludwig,

Since we were scared, we did a full re-init of that specific replica 
from the CA master, and it looks like the issue is not appearing anymore.


Is this sufficient, or should we still investigate ?

Thanks for your help!
Christophe

--

Dr Christophe Trefois, Dipl.-Ing.
Technical Specialist / Post-Doc

UNIVERSITÉ DU LUXEMBOURG

LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE
Campus Belval | House of Biomedicine
6, avenue du Swing
L-4367 Belvaux
T:+352 46 66 44 6124
F:+352 46 66 44 6949
http://www.uni.lu/lcsb

Facebook <https://www.facebook.com/trefex>Twitter 
<https://twitter.com/Trefex>Google Plus 
<https://plus.google.com/+ChristopheTrefois/>Linkedin 
<https://www.linkedin.com/in/trefoischristophe>skype




This message is confidential and may contain privileged information.
It is intended for the named recipient only.
If you receive it in error please notify me and permanently delete 
the original message and any copies.

----


On 18 May 2017, at 16:11, Ludwig Krispenz <lkris...@redhat.com 
<mailto:lkris...@redhat.com>> wrote:


hi,

there was a change that in the case of a missing csn ds would not 
silently use a "close" one and continue, but log an error, backoff 
and retry - after updates on other masters the staring csn coudl 
change and replication continue.


Now, in your case the csn reported missing: 59095fe1000b0012
has a time stamp from May,3rd, so it could very well be correct that 
this csn is no longer found in the changelog.


To continue analysis, could you provide the replicaids of all your 
current replicas, and which is the replicaid of the sever logging 
the change and the ruvs of the replicas from all servers.
ldapsearch   -D "cn=directory manager"  -b cn=config 
"objectclass=nsds5replica" nsds50ruv


Regards,
Ludwig

On 05/18/2017 03:09 PM, Christophe TREFOIS wrote:

Hi all,

Did a yum update on one of my replicas, non CA master, and upgrade 
was successful (ipupgrade.log) said so.



Hwoever, now every few seconds I get the following message. 
https://paste.fedoraproject.org/paste/wS4x9KvD3EB0gv2HAsj6X15M1UNdIGYhyRLivL9gydE=


Does anybody know how to proceed and if this is important?
ipa-replica-manage says, backing off, retrying later, so not sure 
if replication happens successfully or not and what to do ??


Setup: CentOS 7.3 all up-to-date, 2 CA master, 2 non CA master in 
diamond replication.


Remaining replicas were upgraded today as well, and don’t seem to 
complain. Only 1 of them complains.


389-ds-base-libs-1.3.5.10-20.el7_3.x86_64
389-ds-base-1.3.5.10-20.el7_3.x86_64


[root@lums3 ~]# rpm -qa | grep ipa
libipa_hbac-1.14.0-43.el7_3.14.x86_64
python-iniparse-0.4-9.el7.noarch
ipa-admintools-4.4.0-14.el7.centos.7.noarch
python2-ipaserver-4.4.0-14.el7.centos.7.noarch
python2-ipalib-4.4.0-14.el7.centos.7.noarch
sssd-ipa-1.14.0-43.el7_3.14.x86_64
python-ipaddress-1.0.16-2.el7.noarch
python2-ipaclient-4.4.0-14.el7.centos.7.noarch
ipa-server-common-4.4.0-14.el7.centos.7.noarch
ipa-client-common-4.4.0-14.el7.centos.7.noarch
ipa-client-4.4.0-14.el7.centos.7.x86_64
ipa-common-4.4.0-14.el7.ce

Re: [Freeipa-users] Cant locate CSN after yum update

[Ludwig Krispenz]

hi,

there was a change that in the case of a missing csn ds would not 
silently use a "close" one and continue, but log an error, backoff and 
retry - after updates on other masters the staring csn coudl change and 
replication continue.


Now, in your case the csn reported missing: 59095fe1000b0012
has a time stamp from May,3rd, so it could very well be correct that 
this csn is no longer found in the changelog.


To continue analysis, could you provide the replicaids of all your 
current replicas, and which is the replicaid of the sever logging the 
change and the ruvs of the replicas from all servers.
ldapsearch   -D "cn=directory manager"  -b cn=config 
"objectclass=nsds5replica" nsds50ruv


Regards,
Ludwig

On 05/18/2017 03:09 PM, Christophe TREFOIS wrote:

Hi all,

Did a yum update on one of my replicas, non CA master, and upgrade was 
successful (ipupgrade.log) said so.



Hwoever, now every few seconds I get the following message. 
https://paste.fedoraproject.org/paste/wS4x9KvD3EB0gv2HAsj6X15M1UNdIGYhyRLivL9gydE=


Does anybody know how to proceed and if this is important?
ipa-replica-manage says, backing off, retrying later, so not sure if 
replication happens successfully or not and what to do ??


Setup: CentOS 7.3 all up-to-date, 2 CA master, 2 non CA master in 
diamond replication.


Remaining replicas were upgraded today as well, and don't seem to 
complain. Only 1 of them complains.


389-ds-base-libs-1.3.5.10-20.el7_3.x86_64
389-ds-base-1.3.5.10-20.el7_3.x86_64


[root@lums3 ~]# rpm -qa | grep ipa
libipa_hbac-1.14.0-43.el7_3.14.x86_64
python-iniparse-0.4-9.el7.noarch
ipa-admintools-4.4.0-14.el7.centos.7.noarch
python2-ipaserver-4.4.0-14.el7.centos.7.noarch
python2-ipalib-4.4.0-14.el7.centos.7.noarch
sssd-ipa-1.14.0-43.el7_3.14.x86_64
python-ipaddress-1.0.16-2.el7.noarch
python2-ipaclient-4.4.0-14.el7.centos.7.noarch
ipa-server-common-4.4.0-14.el7.centos.7.noarch
ipa-client-common-4.4.0-14.el7.centos.7.noarch
ipa-client-4.4.0-14.el7.centos.7.x86_64
ipa-common-4.4.0-14.el7.centos.7.noarch
python-libipa_hbac-1.14.0-43.el7_3.14.x86_64
ipa-server-4.4.0-14.el7.centos.7.x86_64

Thanks a lot for any pointers,
Christophe

--

Dr Christophe Trefois, Dipl.-Ing.
Technical Specialist / Post-Doc

UNIVERSITÉ DU LUXEMBOURG

LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE
Campus Belval | House of Biomedicine
6, avenue du Swing
L-4367 Belvaux
T:+352 46 66 44 6124
F:+352 46 66 44 6949
http://www.uni.lu/lcsb

Facebook Twitter 
Google Plus 
Linkedin 
skype



This message is confidential and may contain privileged information.
It is intended for the named recipient only.
If you receive it in error please notify me and permanently delete the 
original message and any copies.








--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Replica cannot be reinitialized after upgrade

[Ludwig Krispenz]

The messages you see could be transient messages, and if replication is 
working than this seems to be the case. If not we would need more data 
to investigate: deployment info, relicaIDs of all servers, ruvs, logs,.


Here is some background info: there are some scenarios where a csn could 
not be found in the changelog, eg if updates were aplied on the supplier 
during a total init, they could be part of the data and database ruv, 
but not in the changelog of the initialized replica.
ds did try to use an alternative csn in cases where it could not be 
found, but this had the risk of missing updates, so we decided to change 
it and make this misssing csn a non fatal error, backoff and retry, if 
another supplier would have updated the replica in between, the starting 
csn could have changed and be found. so if the reported missing csns 
change and replication continues everything is ok, although I think the 
messages should stop at some point.


There is a configuration parameter for a replciation agreement to 
trigger the previous behaviour of picking an alternative csn:


nsds5ReplicaIgnoreMissingChange

with potential values "once", "always".

where "once" just tries to kickstart replication by using another csn 
and "always" changes the default behaviour



On 05/11/2017 06:53 PM, Goran Marik wrote:

Hi,

After an upgrade to Centos 7.3.1611 with “yum update", we started seeing the 
following messages in the logs:
“””
May  9 21:58:28 inf01 ns-slapd[4323]: [09/May/2017:21:58:28.519724479 +] 
NSMMReplicationPlugin - changelog program - 
agmt="cn=cloneAgreement1-inf02.dev.ecobee.com-pki-tomcat" (inf02:389): CSN 
576b34e8000a050f not found, we aren't as up to date, or we purged
May  9 21:58:28 inf01 ns-slapd[4323]: [09/May/2017:21:58:28.550459233 +] 
NSMMReplicationPlugin - 
agmt="cn=cloneAgreement1-inf02.dev.ecobee.com-pki-tomcat" (inf02:389): Data 
required to update replica has been purged from the changelog. The replica must be 
reinitialized.
May  9 21:58:32 inf01 ns-slapd[4323]: [09/May/2017:21:58:32.588245476 +] 
agmt="cn=cloneAgreement1-inf02.dev.ecobee.com-pki-tomcat" (inf02:389) - Can't 
locate CSN 576b34e8000a050f in the changelog (DB rc=-30988). If replication stops, 
the consumer may need to be reinitialized.
May  9 21:58:32 inf01 ns-slapd[4323]: [09/May/2017:21:58:32.611400689 +] 
NSMMReplicationPlugin - changelog program - 
agmt="cn=cloneAgreement1-inf02.dev.ecobee.com-pki-tomcat" (inf02:389): CSN 
576b34e8000a050f not found, we aren't as up to date, or we purged
May  9 21:58:32 inf01 ns-slapd[4323]: [09/May/2017:21:58:32.642226385 +] 
NSMMReplicationPlugin - 
agmt="cn=cloneAgreement1-inf02.dev.ecobee.com-pki-tomcat" (inf02:389): Data 
required to update replica has been purged from the changelog. The replica must be 
reinitialized.
“””

The log messages are pretty frequently, every few seconds, and report few 
different CSN numbers that cannot be located.

This happens only on one replica out of 4. We’ve tried "ipa-replica-manage 
re-initialize —from” and “ipa-csreplica-manage re-initialize —from” several times, 
but while both commands report success, the log messages continue to happen. The 
server was rebooted and “systemctl restart ipa” was done few times as well.

The replica seems to be working fine despite the errors, but I’m worried that 
the logs indicate underlaying problem we are not fully detecting. I would like 
to understand better what is triggering this behaviour and how to fix it, and 
if someone else saw them after a recent upgrades.

The software versions are 389-ds-base-1.3.5.10-20.el7_3.x86_64 and 
ipa-server-4.4.0-14.el7.centos.7.x86_64

Thanks,
Goran

--
Goran Marik
Senior Systems Developer

ecobee
250 University Ave, Suite 400
Toronto, ON M5H 3E5





--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Fwd: dirsrv not starting after unplanned outage

[Ludwig Krispenz]

looks like you lost your configuration files dse.ldif and its backup as 
well during the outage.

could you check what you have in /etc/dirsrv/slapd-

you can try to copy one of the *dse.ldif* to dse.ldif and try to 
restart, but that file maybe up to date.


Ludwig

On 05/09/2017 12:00 PM, Bret Wortman wrote:
We had an unplanned power outage which may have affected one of our 
freeipa servers. When trying to start, it now errors out.


# ipactl start
Starting Directory Service
Failed to start Directory Service: Command '/bin/systemctl start 
dirsrv@SPX-NET.service' returned non-zero exit status 1

#

In /var/log/messages, there is a lengthy list of errors like this:

2017-05-09T09:25:40.178252+00:00 asipa ns-slapd: 
[09/May/2017:09:25:40.159091115 +] valueset_value_symtax_cmp: 
slapi_attr_bvalues2keys_sv failed for type attributetypes


ending with:

2017-05-09T09:25:40.178438+00:00 asiopa ns-slapd: 
[09/May/2017:09:25:40:161987520 +] dse_read_one_file - The entry 
cn=schema in file /etc/dirsrv/slapd-DAMASCUSGRP.COM 
/schema/00core.ldif (lineno: 1) is invalid, error code 21 (Invalid 
syntax) - attribute type aci: Unknown attribute syntax OID 
"1.3.6.1.4.1.1466.115.121.1.15"
2017-05-09T09:25:40.178469+00:00 asipa ns-slapd: 
[09/May/2017:09:25:40.162014035 +] dse - Please edit the file to 
correct the reported problems and then restart the server.


The entry in 00core.ldif was verified against the other servers and 
the file hasn't been altered or edited that I can see.


Where else can I look? I've got two servers up, but I'd like to have 
all 3 operational.



--
*Bret Wortman*
Damascus Products
ph/fax: 1-855-644-2783
Wrap Buddies InDemand at http://bwortman.us/2ieQN4t




--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] LDAP Conflicts

[Ludwig Krispenz]

you can start here:

https://access.redhat.com/documentation/en-us/red_hat_directory_server/10/html/administration_guide/managing_replication-solving_common_replication_conflicts

you need first find out which conflict entries you have, which entries 
need to be preserved, and then can start to rename or delete the 
conflicts. there is no magic tool.


On 05/04/2017 04:20 PM, James Harrison wrote:

Hello All,
According to ipa_check_consistency we have "LDAP Conflicts" 
(https://github.com/peterpakos/ipa_check_consistency).


How do I find and resolve them?

I've seen:
Re: [Freeipa-devel] LDAP conflicts resolution API 







Re: [Freeipa-devel] LDAP conflicts resolution API





But not sure if I am looking in the right place.

Many thanks,
James Harrison




--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] consumer replica which does not show up in ruv list

[Ludwig Krispenz]

On 03/07/2017 09:21 PM, lejeczek wrote:



On 07/03/17 16:48, Ludwig Krispenz wrote:


On 03/07/2017 05:29 PM, lejeczek wrote:



On 07/03/17 12:39, Martin Babinsky wrote:

On Tue, Mar 07, 2017 at 09:55:52AM +, lejeczek wrote:

hi,

I presume I need to use ldapmodify/delete?
I found this(obfuscated by me):

cn=dzien.priv.xx.xx.priv.xx.xx.x+nsuniqueid=9e47680e-296e11e6-83a59f45-6ec26a1e,cn=masters,cn=ipa,cn=etc,dc=priv,dc=xx.dc=xx.dc=priv,dc=xx,dc=xx,dc=x 



To confirm? Would removing it fix the problem? I'm probably 
missing something

else, aren't I?

many thank,
L
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project


That seems like a replication conflict. Consult the following guide 
to solve

it:

https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/10/html/Administration_Guide/Managing_Replication-Solving_Common_Replication_Conflicts.html 

I'm not sure whether I'm dealing with single or multi-valued DN and 
I should rename+keep original copy(following that doc) or simply 
remove that DN.
this is something which cannot be generally answered, you need to 
look at the specific entries. In the case of conflicts you always 
have entries like

cn=, and
cn=+nsuniqueid=-nnn-nnn-nn,

and usually they are created if the same entry is added at the same 
time on two replicas, then they are identical and you can just delete 
the conflict entry. Only if you want to keep both entries you need to 
rename the conflict.


to confirm - I presume this should be a recursive deletion with '-r' , 
the whole lot, right?
I would not recommend this before examining what the "whole lot", you 
need to spend a bit time in examining the state of your data, if there 
are nodes below the conflict entries  are tehy conflicts as well, can 
they be deleted or should they be moved to the valid entry first,  ?

thx,
L.



Just a side question, how did you end up with such entry? Did you 
happen to upgrade

multiple IPA masters at once?









--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] consumer replica which does not show up in ruv list

[Ludwig Krispenz]

On 03/07/2017 05:29 PM, lejeczek wrote:



On 07/03/17 12:39, Martin Babinsky wrote:

On Tue, Mar 07, 2017 at 09:55:52AM +, lejeczek wrote:

hi,

I presume I need to use ldapmodify/delete?
I found this(obfuscated by me):

cn=dzien.priv.xx.xx.priv.xx.xx.x+nsuniqueid=9e47680e-296e11e6-83a59f45-6ec26a1e,cn=masters,cn=ipa,cn=etc,dc=priv,dc=xx.dc=xx.dc=priv,dc=xx,dc=xx,dc=x 



To confirm? Would removing it fix the problem? I'm probably missing 
something

else, aren't I?

many thank,
L
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project


That seems like a replication conflict. Consult the following guide 
to solve

it:

https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/10/html/Administration_Guide/Managing_Replication-Solving_Common_Replication_Conflicts.html 

I'm not sure whether I'm dealing with single or multi-valued DN and I 
should rename+keep original copy(following that doc) or simply remove 
that DN.
this is something which cannot be generally answered, you need to look 
at the specific entries. In the case of conflicts you always have 
entries like

cn=, and
cn=+nsuniqueid=-nnn-nnn-nn,

and usually they are created if the same entry is added at the same time 
on two replicas, then they are identical and you can just delete the 
conflict entry. Only if you want to keep both entries you need to rename 
the conflict.


Just a side question, how did you end up with such entry? Did you 
happen to upgrade

multiple IPA masters at once?





--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] replication breaks intermittently

[Ludwig Krispenz]

On 03/01/2017 08:18 PM, pgb205 wrote:
[01/Mar/2017:18:19:48 +] agmt="cn=meTo ipa2.internal.domain" 
(ipa2:389) - Can't locate CSN 582301c3000d0077 in the changelog 
(DB rc=-30988). If replication stops, the consumer may need to be 
reinitialized.
[01/Mar/2017:18:19:48 +] NSMMReplicationPlugin - changelog program 
- agmt="cn=meToipa2.internal.domain" (ipa2:389): CSN 
582301c3000d0077 not found, we aren't as up to date, or we purged
[01/Mar/2017:18:19:48 +] NSMMReplicationPlugin - 
agmt="cn=meToipa2.internal.domain" (ipa2:389): Data required to update 
replica has been purged. The replica must be reinitialized.
[01/Mar/2017:18:19:48 +] NSMMReplicationPlugin - 
agmt="cn=meToipa2.internal.domain" (ipa2:389): Incremental update 
failed and requires administrator action
the timestamp in the csn: 582301c3 is from Nov,9, 2016. so it is very 
likely that it was trimmed from the changelog or lost during some 
reinitialization.
could you verify to which server the replicaid '0077' == 119 refers and 
if it still exists





Seeing the above in the logs after seeing problems with replication. 
The data entered on one replica isn't making it to others. There are 
no problems with connectivity between the servers and all necessary 
ports are open. Currently we are getting around this problem by 
running a script to perform force sync.


Any suggestions on the things that may be setup wrong to take a look at?






--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] unable to decode: {replica

[Ludwig Krispenz]

On 02/28/2017 07:52 PM, lejeczek wrote:



On 28/02/17 09:45, Petr Vobornik wrote:

On 02/26/2017 11:35 AM, lejeczek wrote:

hi everyone

I first time see:

unable to decode: {replica 60} 586eaffd000a003c 
586eaffd000a003c

Replica Update Vectors:


on all four servers. What would be a correct troubleshooting and 
fixing this

problem?
many thanks,
L.





Hello,

what is the version and OS of your IPA servers and DS?

 $ rpm -q ipa-server freeipa-server 389-ds-base

well I run a Centos 7.x and
~]$ rpm -q ipa-server freeipa-server 389-ds-base
ipa-server-4.4.0-14.el7.centos.4.x86_64
package freeipa-server is not installed
389-ds-base-1.3.5.10-15.el7_3.x86_64

I searched the net and archives but failed to find anything flagged as 
"solved".
if you expect help, you should provide a bit more information than the 
snippet of an error message. As Petr pointed out this looks like a 
problem of a corrupted RUV, but we also haven't seen these for a long time.
Could you describe your deployment, what changed recently 
(addigng/removing replicas, crashes, )

A mapping of servers and replica Ids, to which server does "60" refer?
Check the ruvs for all suffixes on all servers.
Try cleaning the RUV, if IPA command does not work do it by ldapmodify

There have been many discussions on this topic in this mailing list, 
look for "cleanallruv", "haunted servers",..


Ludwig





Similar issues happened last year, you can search the archives for 
"unable to decode" but a 389-ds fix improved the situation. So if you 
have older version then maybe update and then manual cleanup of RUVs 
might help.








--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] how to resolve replication conflicts

[Ludwig Krispenz]

On 02/16/2017 01:32 PM, Tiemen Ruiten wrote:

Hello,

I have a FreeIPA setup in which some masters suffered from a few 
uncontrolled shutdowns and now there are replication conflicts (which 
prevent from setting the Domain Level to 1).


I was trying to follow the instructions here: 
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Identity_Management_Guide/ipa-replica-manage.html


But unfortunately I'm not getting anywhere. This the result of an 
ldapsearch for replication conflicts:



[root@moscovium ~]# ldapsearch -x -D "cn=directory manager" -W -b
"dc=ipa,dc=rdmedia,dc=com" "nsds5ReplConflict=*" \* nsds5ReplConflict
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base 

Re: [Freeipa-users] be_pam_handler_callback Backend returned: (3, 4, ) [Internal Error (System error)]

[Ludwig Krispenz]

thanks for the info

Ludwig

On 01/20/2017 11:43 AM, Harald Dunkel wrote:

On 01/19/17 16:23, Harald Dunkel wrote:

Now I get this:

[root@ipa1 ~]# kinit admin
kinit: Generic error (see e-text) while getting initial credentials


Fortunately this went away after a reboot of the servers.

Phew
Harri



--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] be_pam_handler_callback Backend returned: (3, 4, ) [Internal Error (System error)]

[Ludwig Krispenz]

On 01/18/2017 02:57 PM, Harald Dunkel wrote:

On 01/17/17 11:38, Sumit Bose wrote:

On Tue, Jan 17, 2017 at 10:44:14AM +0100, Harald Dunkel wrote:

It seems something got corrupted in my ipa setup. I found this in the
sssd log file on Wheezy:

(Tue Jan 17 10:19:02 2017) [hbac_shost_attrs_to_rule] (0x0400): Processing 
source hosts for rule [allow_all]
(Tue Jan 17 10:19:02 2017) [hbac_eval_user_element] (0x0080): Parse error on 
[cn=System: Manage Host 
Principals+nsuniqueid=109be36e-ccd911e6-a5b3d0c8-d8da17db,cn=permissions,cn=pbac,dc=example,dc=de]

Looks like there was a replication conflict, please see
https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/10/html/Administration_Guide/Managing_Replication-Solving_Common_Replication_Conflicts.html
how to resolve it.


This is *way* too hot for me.
I think the procedure in the link about renaming is only needed if you 
want to keep both entries with a "normal" dn. But you want to get rid of 
the conflict entries.  Since you have to cleanup each of them 
individually I would suggest to start with one of them.


First get both the conflict entry and the normal entry and compare them:
ldapsearch   -D "cn=directory manager" . -b "cn=System: Manage Host 
Principals,cn=permissions,cn=pbac,dc=example,dc=de" -s base
ldapsearch  -D "cn=directory manager"  . -b "cn=System: Manage Host 
Principals+nsuniqueid=109be36e-ccd911e6-a5b3d0c8-d8da17db,cn=permissions,cn=pbac,dc=example,dc=de" 
-s base


They should be identical.
Next check if the conflict entry has child entries:
ldapsearch  -D "cn=directory manager"  . -b "cn=System: Manage Host 
Principals+nsuniqueid=109be36e-ccd911e6-a5b3d0c8-d8da17db,cn=permissions,cn=pbac,dc=example,dc=de" 
dn


If there are no entries below the conflict entry you can remove it:
ldapmodify - D "cn=directory manager" ..
dn: cn=System: Manage Host 
Principals+nsuniqueid=109be36e-ccd911e6-a5b3d0c8-d8da17db,cn=permissions,cn=pbac,dc=example,dc=de

changetype: delete


How can I try this in a sandbox?

you can try to reproduce this state on two other machines.
and if you have an established backup and restore process do a backup 
before doing the cleanup



Every helpful comment is highly appreciated
Harri



--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] be_pam_handler_callback Backend returned: (3, 4, ) [Internal Error (System error)]

[Ludwig Krispenz]

On 01/18/2017 08:13 AM, Harald Dunkel wrote:

Hi Ludwig,

On 01/17/17 17:01, Ludwig Krispenz wrote:

On 01/17/2017 04:48 PM, Harald Dunkel wrote:

On 01/17/17 16:12, Harald Dunkel wrote:

On 01/17/17 11:38, Sumit Bose wrote:

On Tue, Jan 17, 2017 at 10:44:14AM +0100, Harald Dunkel wrote:

It seems something got corrupted in my ipa setup. I found this in the
sssd log file on Wheezy:

(Tue Jan 17 10:19:02 2017) [hbac_shost_attrs_to_rule] (0x0400): Processing 
source hosts for rule [allow_all]
(Tue Jan 17 10:19:02 2017) [hbac_eval_user_element] (0x0080): Parse error on 
[cn=System: Manage Host 
Principals+nsuniqueid=109be36e-ccd911e6-a5b3d0c8-d8da17db,cn=permissions,cn=pbac,dc=example,dc=de]

Looks like there was a replication conflict, please see
https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/10/html/Administration_Guide/Managing_Replication-Solving_Common_Replication_Conflicts.html
how to resolve it.


% ldapsearch -D "cn=directory manager" -w secret -b "dc=example,dc=de" 
"nsds5ReplConflict=*" \* nsds5ReplConflict | grep nsds5ReplConflict | wc -l
26


PS:

nsds5ReplConflict: namingConflict 
cn=ipaservers,cn=hostgroups,cn=accounts,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=ipaservers,cn=ng,cn=alt,dc=example,dc=de
nsds5ReplConflict: namingConflict 
cn=domain,cn=topology,cn=ipa,cn=etc,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=locations,cn=etc,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=dns 
administrators,cn=privileges,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=dns 
servers,cn=privileges,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=cas,cn=ca,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=custodia,cn=ipa,cn=etc,dc=example,dc=de
nsds5ReplConflict: namingConflict 
cn=dogtag,cn=custodia,cn=ipa,cn=etc,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=system: add 
ca,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=system: delete 
ca,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=system: modify 
ca,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=system: read 
cas,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=system: modify dns servers 
configuration,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=system: read dns servers 
configuration,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=System: Manage Host 
Principals,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=System: Add IPA 
Locations,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=System: Modify IPA 
Locations,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=System: Read IPA 
Locations,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=System: Remove IPA 
Locations,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=System: Read Locations of IPA 
Servers,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=System: Read Status of Services on IPA 
Servers,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=System: Manage Service 
Principals,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=System: Manage User 
Principals,cn=permissions,cn=pbac,dc=example,dc=de

This looks like a problem of ipa-server-install. These entries were created
in the very first seconds.

Conflict entries are created if an entry is added on different servers at the "same 
time", where same time means it is created on instance x before the add of the entry 
on instance y was replicated to x. This can happen if you run things in parallel, eg 
upgrades.


You mean Freeipa has a race condition? I use tools like clusterssh to
install or upgrade several hosts in parallel (n <= 49 due to available
screen and font size). The "same time" is built in.

Of course I understand that Freeipa is a special case, because it is
network application, but it should be able to handle n = 2.
it is handling it (389-ds does), but the way it is handling it has the 
conflict entries as a result.


If you add an entry to an instance it is assigned a unique identifier, 
the nsuniqeuid. The dn of an entry can change by modrdn, but the 
nsuniqueid is not changed. The nsuniqueid is also used in replication to 
identify which is the entry the replicated op has to be applied to.If 
you add entries with the same dn in parallel you have entries with the 
same dn but different nsuniqueids, this cannot be prevented, on instance 
has no control of what happens on another instance.
But if these entries are replicated you have two entries with the same 
dn but different nsuniqueids on the same server. This violates the ldap 
data model and ds needs to handle it. The way we do it is to 

Re: [Freeipa-users] be_pam_handler_callback Backend returned: (3, 4, ) [Internal Error (System error)]

[Ludwig Krispenz]

On 01/17/2017 04:48 PM, Harald Dunkel wrote:

On 01/17/17 16:12, Harald Dunkel wrote:

On 01/17/17 11:38, Sumit Bose wrote:

On Tue, Jan 17, 2017 at 10:44:14AM +0100, Harald Dunkel wrote:

It seems something got corrupted in my ipa setup. I found this in the
sssd log file on Wheezy:

(Tue Jan 17 10:19:02 2017) [hbac_shost_attrs_to_rule] (0x0400): Processing 
source hosts for rule [allow_all]
(Tue Jan 17 10:19:02 2017) [hbac_eval_user_element] (0x0080): Parse error on 
[cn=System: Manage Host 
Principals+nsuniqueid=109be36e-ccd911e6-a5b3d0c8-d8da17db,cn=permissions,cn=pbac,dc=example,dc=de]

Looks like there was a replication conflict, please see
https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/10/html/Administration_Guide/Managing_Replication-Solving_Common_Replication_Conflicts.html
how to resolve it.


% ldapsearch -D "cn=directory manager" -w secret -b "dc=example,dc=de" 
"nsds5ReplConflict=*" \* nsds5ReplConflict | grep nsds5ReplConflict | wc -l
26


PS:

nsds5ReplConflict: namingConflict 
cn=ipaservers,cn=hostgroups,cn=accounts,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=ipaservers,cn=ng,cn=alt,dc=example,dc=de
nsds5ReplConflict: namingConflict 
cn=domain,cn=topology,cn=ipa,cn=etc,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=locations,cn=etc,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=dns 
administrators,cn=privileges,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=dns 
servers,cn=privileges,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=cas,cn=ca,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=custodia,cn=ipa,cn=etc,dc=example,dc=de
nsds5ReplConflict: namingConflict 
cn=dogtag,cn=custodia,cn=ipa,cn=etc,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=system: add 
ca,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=system: delete 
ca,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=system: modify 
ca,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=system: read 
cas,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=system: modify dns servers 
configuration,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=system: read dns servers 
configuration,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=System: Manage Host 
Principals,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=System: Add IPA 
Locations,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=System: Modify IPA 
Locations,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=System: Read IPA 
Locations,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=System: Remove IPA 
Locations,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=System: Read Locations of IPA 
Servers,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=System: Read Status of Services on IPA 
Servers,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=System: Manage Service 
Principals,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=System: Manage User 
Principals,cn=permissions,cn=pbac,dc=example,dc=de

This looks like a problem of ipa-server-install. These entries were created
in the very first seconds.
Conflict entries are created if an entry is added on different servers 
at the "same time", where same time means it is created on instance x 
before the add of the entry on instance y was replicated to x. This can 
happen if you run things in parallel, eg upgrades.


There is no simple way to get rid of them, you need to delete them one 
by one, so do:

ldapmodify ...
dn:  cn=System: Manage Host 
Principals+nsuniqueid=109be36e-ccd911e6-a5b3d0c8-d8da17db,cn=permissions,cn=pbac,dc=example,dc=de

changetype: delete

for all of your conflict entries



Harri



--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] SLAPD stops answering

[Ludwig Krispenz]

Hi,

there seem to be to issues here, maybe related: a hanging slapd process 
and the retro CL errors.


If the slapd process is not responding can we get a pstack or gdb 
backtrace (http://www.port389.org/docs/389ds/FAQ/faq.html#debug_crashes) 
of the process ?
About the Retro CL messages, is it always the same changenumber which 
is  reported ?


On 01/09/2017 02:06 PM, Troels Hansen wrote:

Hi, we have a IPA installation, which obviously needs upgrading.
Its a single server running RHEL7.1 running IPA 4.1

However, it have been running smooth untill now:

Rebooting makes everything running again, but only for a few days.

It looks like everything fails around 0:17:47 and comes up again just 
before 8, when the server is rebooted.


Jan  6 00:19:46 fbbidm01 winbindd[2965]: failed to bind to server 
ldapi://%2fvar%2frun%2fslapd-DOMAIN.LAN.socket with dn="[Anonymous 
bind]" Error: Local error

Jan  6 00:19:46 fbbidm01 winbindd[2965]: (unknown)
Jan  6 00:20:29 fbbidm01 winbindd[2965]: [2017/01/06 00:20:29.758332,  
0] ipa_sam.c:4128(bind_callback_cleanup)


Looking at the SLAPD logs also reveals it stopped answering:

[06/Jan/2017:00:17:47 +0100] conn=40702 op=62 SRCH 
base="cn=radius_aura_admin,cn=groups,cn=accounts,dc=domain,dc=lan" 
scope=0 filter="(objectClass=*)" attrs="cn"
[06/Jan/2017:00:17:47 +0100] conn=40702 op=62 RESULT err=0 tag=101 
nentries=1 etime=0
[06/Jan/2017:00:17:47 +0100] conn=40702 op=63 SRCH 
base="cn=radius_users,cn=groups,cn=accounts,dc=domain,dc=lan" scope=0 
filter="(objectClass=*)" attrs="cn"
[06/Jan/2017:00:17:47 +0100] conn=40702 op=63 RESULT err=0 tag=101 
nentries=1 etime=0
[06/Jan/2017:00:17:47 +0100] conn=40702 op=64 SRCH 
base="cn=system_radius_users,cn=groups,cn=accounts,dc=domain,dc=lan" 
scope=0 filter="(objectClass=*)" attrs="cn"
[06/Jan/2017:00:17:47 +0100] conn=40702 op=64 RESULT err=0 tag=101 
nentries=1 etime=0
[06/Jan/2017:00:17:48 +0100] conn=40702 op=65 SRCH 
base="cn=accounts,dc=domain,dc=lan" scope=2 
filter="(uid=sys_prov_aura)" attrs=ALL
[06/Jan/2017:00:17:48 +0100] conn=40702 op=65 RESULT err=0 tag=101 
nentries=1 etime=0
[06/Jan/2017:00:17:48 +0100] conn=40702 op=66 BIND 
dn="uid=sys_prov_aura,cn=users,cn=accounts,dc=domain,dc=lan" 
method=128 version=3
[06/Jan/2017:00:17:48 +0100] conn=40702 op=66 RESULT err=0 tag=97 
nentries=0 etime=0 
dn="uid=sys_prov_aura,cn=users,cn=accounts,dc=domain,dc=lan"
[06/Jan/2017:00:17:51 +0100] conn=40703 fd=158 slot=158 connection 
from 10.250.8.66 to 10.250.8.58
[06/Jan/2017:00:17:53 +0100] conn=40704 fd=159 slot=159 SSL connection 
from 10.250.8.37 to 10.250.8.58
[06/Jan/2017:00:18:02 +0100] conn=40705 fd=160 slot=160 SSL connection 
from 10.250.8.57 to 10.250.8.58
[06/Jan/2017:00:18:02 +0100] conn=40706 fd=161 slot=161 SSL connection 
from 10.250.20.102 to 10.250.8.58
[06/Jan/2017:00:18:03 +0100] conn=40707 fd=162 slot=162 SSL connection 
from 10.250.20.102 to 10.250.8.58
[06/Jan/2017:00:18:58 +0100] conn=40708 fd=163 slot=163 connection 
from 10.250.8.66 to 10.250.8.58
[06/Jan/2017:00:19:03 +0100] conn=40709 fd=164 slot=164 connection 
from local to /var/run/slapd-DOMAIN.LAN.socket
[06/Jan/2017:00:19:35 +0100] conn=40710 fd=165 slot=165 connection 
from 10.250.8.58 to 10.250.8.58
[06/Jan/2017:00:19:35 +0100] conn=40711 fd=166 slot=166 connection 
from 10.150.27.7 to 10.250.8.58
[06/Jan/2017:00:19:43 +0100] conn=40712 fd=167 slot=167 SSL connection 
from 10.250.20.102 to 10.250.8.58
[06/Jan/2017:00:19:46 +0100] conn=40713 fd=168 slot=168 connection 
from local to /var/run/slapd-DOMAIN.LAN.socket


It looks like it just stops answering at 00:17:48

The slapd error log reveals nothing:

[06/Jan/2017:00:17:34 +0100] DSRetroclPlugin - replog: an error 
occured while adding change number 3875312, dn = 
changenumber=3875312,cn=changelog: Already exists.
[06/Jan/2017:00:17:34 +0100] retrocl-plugin - retrocl_postob: 
operation failure [68]
[06/Jan/2017:07:57:21 +0100] - slapd shutting down - signaling 
operation threads - op stack size 0 max work q size 735 max work q 
stack size 23
[06/Jan/2017:07:57:21 +0100] - slapd shutting down - waiting for 30 
threads to terminate
[06/Jan/2017:07:58:02 +0100] SSL Initialization - Configured SSL 
version range: min: TLS1.0, max: TLS1.2


However, see a gazillion of these lines in the error log:

DSRetroclPlugin - replog: an error occured while adding change number 
3875312, dn = changenumber=3875312,cn=changelog: Already exists.


Anyone with some thoughts about this, other that "Just upgrade".

--

Med venlig hilsen

*Troels Hansen*

Systemkonsulent

Casalogic A/S

T  (+45) 70 20 10 63

M (+45) 22 43 71 57

 
 
Red Hat, SUSE, VMware, Citrix, Novell, Yellowfin BI, EnterpriseDB, 
Sophos og meget mere.





--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael 

Re: [Freeipa-users] FreeIPA 4.4 - Can't find topology segment, nsunique attribute

[Ludwig Krispenz]

Hi
On 12/22/2016 09:31 AM, Georgijs Radovs wrote:

Hello everyone!

Today, I've updated 2 FreeIPA servers from version 4.2 to version 4.4.

Both of these servers are Masters and CAs, both are replicating 
between each other.


But, when I run

*ipa topologysegment-find* to view replication agreements for *domain* 
and *ca* suffixes


it returns zero results.

Web UI also does not show any agreements, but when I try to create a 
replication agreement between both servers, I get error that agreement 
already exists.


Also, when viewing directory using ldap browser, I found these 
containers:


DN: 
cn=ca+nsuniqueid=7252d047-c76611e6-a1fcaefe-5d4473a3,cn=topology,cn=ipa,cn=etc,dc=example,dc=com


DN: 
cn=domain+nsuniqueid=7252d000-c76611e6-a1fcaefe-5d4473a3,cn=topology,cn=ipa,cn=etc,dc=example,dc=com


Both of them contain topology segments, which I'm trying to create, 
but they do not show up anywhere.
this is unfortunatly the result of raising the domainlevel and creating 
segments while replication conflict entries exist.
In the next update we will prevent this by checking for conflicts before 
raising domainlevel.


How do I remove nsuniqueid attribute or delete those containers?
not so simple, I'll try to sketch the options for cn=domain, the 
procedure for cn=ca is then the same.


So lets say you have:
cn=domain,cn=topology,cn=ipa,cn=etc,dc=example,dc=com
cn=domain+nsuniqueid=7252d000-c76611e6-a1fcaefe-5d4473a3,cn=topology,cn=ipa,cn=etc,dc=example,dc=com
cn=segment1,cn=domain+nsuniqueid=7252d000-c76611e6-a1fcaefe-5d4473a3,cn=topology,cn=ipa,cn=etc,dc=example,dc=com

and what you want is:
cn=domain,cn=topology,cn=ipa,cn=etc,dc=example,dc=com
cn=segment1,cn=domain,cn=topology,cn=ipa,cn=etc,dc=example,dc=com

unfortunately the segment is below the conflict entry. so you have two 
options:
- remove the "normal" entry and then rename the conflict entry, this 
will leave child parent relationship and the dn of the segment should be 
adjusted automatically


- move the segment to the "normal" entry (it could be rejected by the 
topology plugin, so you would have to disable it temporariliy on the 
server where you run this)

and then remove the "conflict" entry

--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] modify schema - add group email and display attribute

[Ludwig Krispenz]

On 12/21/2016 02:07 PM, Sandor Juhasz wrote:

Hi,

i would like to modify schema to have group objects extended with 
email and display name attribute.

The reason is that we are trying to sync our ldap to our google apps.

I don't know how much this 
doc http://www.freeipa.org/images/5/5b/FreeIPA33-extending-freeipa.pdf
can be applied to groups. Neither did i find a supported attribute 
syntax for email, maybe
PrintableString 	1.3.6.1.4.1.1466.115.121.1.58 	For values which 
contain strings containing alphabetic, numeral, and select punctuation 
characters (as defined in RFC 4517 ).


but i am not sure if that could hold email addresses.
why don't you just use the mail attribute ? only define a new auxilliary 
objectclass allowing mail and displayname


It would be pretty to have it exposed via ipalib and js plugins as well.
If someone could help me out on extending schema, i would be really happy.

*Sándor Juhász*
System Administrator
*ChemAxon**Ltd*.
Building Hx, GraphiSoft Park, Záhony utca 7, Budapest, Hungary, H-1031
Cell: +36704258964




--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] freeipa 4.1 replication conflict resolve issue

[Ludwig Krispenz]

On 12/21/2016 05:11 AM, Ian Chen wrote:

hello list,

I tried to search for answer, but not solution come up yet. please help.

the setup with multiple nodes has IPA version:
ipa-server-4.1.0-18.el7.centos.4.x86_64


after adding a replication with an old node, replicaiton conflict occured.

 node104
dn: 
nsuniqueid=5820a804-af9211e6-bbce8d9c-0794b841+uid=test2,cn=users,cn=acco

 unts,dc=...
uid: test2
nsds5ReplConflict: namingConflict uid=test2,cn=users,cn=accounts,dc=...
krbPrincipalName: test2@...
krbLastPwdChange: 20161220054653Z
krbPasswordExpiration: 20170320054653Z
ipaUniqueID: 606b2260-af92-11e6-a928-0050568faf9d


 node203
dn: uid=test2,cn=users,cn=accounts,dc=...
uid: test2
krbPrincipalName: test2@...
krbLastPwdChange: 20161220054653Z
krbPasswordExpiration: 20170320054653Z
ipaUniqueID: 606b2260-af92-11e6-a928-0050568faf9d


I tried rename RDN following this
https://mkosek.fedorapeople.org/publican_site/en-US/FreeIPA/3.4/html/FreeIPA_Guide/ipa-replica-manage.html

but when trying to delete uid, then change RDN back to uid, there is 
this error


modifying entry "cn=TempValue,cn=users,cn=accounts,dc=..."
ldap_modify: Object class violation (65)
additional info: missing attribute "uid" required by object class 
"posixAccount"


I cannot delete object class posixAccount then add it back
I cannot see which commands you really tried to execute and failed, so 
could you provide the full log of what you did if you want to follow the 
steps in the IPA doc.


But I do not think that you need to go thru the MOD/MODRDN/... sequence 
if you do not want to keep both entries. If a conflict arises, one entry 
keeps the original dn, the other gets a dn with "nsuniquid=+..." and 
the nsds5ReplConflict attribute. you can check the entries and inmost 
cases you just want to keep the "original" and just delete the conflict 
entry





--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] ipa-replica-install fails because dirsrv failed to start

[Ludwig Krispenz]

On 10/27/2016 10:48 AM, Jochen Demmer wrote:



Am 27.10.2016 um 10:21 schrieb Martin Basti:




On 27.10.2016 10:02, Jochen Demmer wrote:



Am 26.10.2016 um 17:31 schrieb Martin Basti:




On 26.10.2016 17:25, Jochen Demmer wrote:



Am 26.10.2016 um 16:48 schrieb Martin Basti:




On 26.10.2016 16:42, Jochen Demmer wrote:



Am 26.10.2016 um 16:27 schrieb Martin Basti:




On 26.10.2016 16:10, Jochen Demmer wrote:

Hi,

my answers also inline.

Am 26.10.2016 um 15:38 schrieb Martin Basti:


Hi, comments inline


On 26.10.2016 14:28, Jochen Demmer wrote:

Hi,

I've been running and using a single FreeIPA server 
successfully, i.e.:

Fedora 24
freeipa-server-4.3.2-2.fc24.x86_64
This server is only available via IPv6, because I can't get 
public lPv4 addresses no more.


Now I want to setup a FreeIPA replica at another site also 
running IPv6, Fedora 24 and freeipa-server-4.3.2-2.fc24.x86_64
First I run "ipa-client-install" which succeeds without an 
error.

When I invoke "ipa-replica-install" I get this error:
ipa : ERRORCould not resolve hostname 
*hostname.mydoma.in* using DNS. Clients may not function 
properly. Please check your DNS setup. (Note that this check 
queries IPA DNS directly and ignores /etc/hosts.)

LOG:
2016-10-26T12:14:39Z DEBUG Search DNS server 
*hostname.mydoma.in* (['2a01:f11:1:1::1', '2a01:f11:1:1::1', 
'2a01:f11:1:1::1']) for *hostname.mydoma.in*


Can you check with dig or host command if the hostname is 
really resolvable on that machine? do you have proper 
resolver in /etc/resolv.conf?
There is a resolver given in /etc/resolv.conf. When I do "host 
<>" I get the right IPv6 back.

That is weird because IPA is doing basically the same.





*hostname.mydoma.in* is actually the DNS entry for the old 
FreeIPA server, which actually resolves, but only to an IPv6 
address of course.

I can continue the installation though by entering "yes".

I then get asked:
Enter the IP address to use, or press Enter to finish.
Please provide the IP address to be used for this host name:

When I enter the IPv6 address of the new replica host it 
doesn't accept but infinitely asks this question instead.


Have you pressed enter twice? It should end prompt and 
continue with installation

Enter without an IP -> No usable IP address provided nor resolved.
Enter with an IP -> Error: Invalid IP Address 2a02:1:2:3::4 
cannot use IP network address 2a02:1:2:3::4 


How do you have configured IP address on your interface? Does 
it have prefix /128?
Yes, that's right. It's an IP being assigned statefully by a 
DHCPv6 server.
There is also another dynamic IP within the same prefix having 
/64. I don't want to use this one of course, because its IID 
changes.


Could you set (temporarily) prefix for that address to /64 and 
re-run installer? IPA 4.3 has check that prevents you to use /128 
prefix
Well now I don't even get asked for the IP. The setup wizard 
continues, but I now get this error:


  [27/43]: restarting directory server
ipa : CRITICAL Failed to restart the directory server 
(Command '/bin/systemctl restart dirsrv@MY-REALM.service' returned 
non-zero exit status 1). See the installation log for details.

  [28/43]: setting up initial replication
  [error] error: [Errno 111] Connection refused

LOG:
2016-10-26T15:14:46Z DEBUG Process finished, return code=1
2016-10-26T15:14:46Z DEBUG stdout=
2016-10-26T15:14:46Z DEBUG stderr=Job for dirsrv@MY-REALM.service 
failed because the control process exited with error code. See 
"systemctl status dirsrv@MY-REALM.service" and "journalctl -xe" 
for details.
2016-10-26T15:14:46Z CRITICAL Failed to restart the directory 
server (Command '/bin/systemctl restart dirsrv@MY-REALM.service' 
returned non-zero exit status 1). See the installation log for 
details.

2016-10-26T15:14:46Z DEBUG   duration: 1 seconds
2016-10-26T15:14:46Z DEBUG   [28/43]: setting up initial replication
2016-10-26T15:14:56Z DEBUG Traceback (most recent call last):

When I try to restart manually with, "/bin/systemctl restart 
dirsrv@MY-REALM.service"

 this is what systemd logs:
https://paste.fedoraproject.org/461439/raw/




Could you please check /var/log/dirsrv/slapd-*/errors  there might 
be more details.


Did you reused an old IPA server for this installation?

Martin

This is what the logfile says:
https://paste.fedoraproject.org/461685/raw/

I tried to install this server as a replica a couple of times, but I 
even reinstalled all of the software and I keep using

ipa-client-install --uninstall and
ipa-server-install --uninstall


It looks that DS database is somehow corrupted, is possible that 
there might be some leftovers from previous installations


start: Failed to start databases, err=-1 BDB0092 Unknown error: -1

I'm not sure what that error means, maybe DS guys will know

Can you run server uninstall twice? It should remove all leftovers, 
and then check /var/lib/dirsrv/ if there are any slapd-* directories, 
if yes please remove them


Martin
I 

Re: [Freeipa-users] Replica Problem (Errors)

[Ludwig Krispenz]

On 10/24/2016 01:21 PM, Günther J. Niederwimmer wrote:

Hello Ludwig,

thanks for the answer,


Am Montag, 24. Oktober 2016, 09:53:21 schrieb Ludwig Krispenz:

On 10/23/2016 03:01 PM, Günther J. Niederwimmer wrote:

I have added on my ipa (Master) Server this user and ACI with a ldif file

ldapmodify -x -D 'cn=Directory Manager' -W
dn: uid=system,cn=sysaccounts,cn=etc,dc=example,dc=com
changetype: add
objectclass: account
objectclass: simplesecurityobject
uid: system
userPassword: secret123
passwordExpirationTime: 20380119031407Z
nsIdleTimeout: 0

^D

dn: cn=users,cn=accounts,dc=example,dc=com
changetype: modify
add: aci
aci: (targetattr="mailAlternateAddress")
(targetfilter="(objectClass=mailrecipient)")

(version
3.0; acl "Allow system account to read mail address"; allow(read,
search, compare) userdn =
"ldap:///uid=system,cn=sysaccounts,cn=etc,dc=example,dc=com;;)

This Ends with a
modifying entry "cn=users,cn=accounts,dc=example,dc=com"

these changes are not related to the errors you report below (I would be
really surprised) and you only need to apply them on one server, that's
what replication is good for.

There are a couple of different types of messages:
- failed to delete changelog record: this is from retro changelog
trimming, when miscalculation of the starting point for trimming starts
with changenumber lower than what's in the retro changelog.
In my experience this can happen after a crash/kill/reboot and should
stop after som time

OK, nothing to do ;-).
  

- attrlist_replace errors: looks like you have recreated a replica on a
machine and not cleaned the RUV, please see:
http://www.freeipa.org/page/Troubleshooting#Obsolete_RUV_records

I don't have add or remove a replica ? this two servers running now I mean
over three month ?
that is strange, could you perform step 1] and 2] of this recipe: 
https://www.redhat.com/archives/freeipa-users/2016-May/msg00043.html

but add the option "-o ldif-wrap=no" to the ldapsearch to get the full ruv


The last I remember I add a 3rd Party Certificate ?

but I don't found before so much Errors :-(.

Is there a possible way to check a freeIPA Installation, to find out for a
"normal" user to have a consistent System ?


- keep-alive already exists: this is also an indication of a new
replica, the keep alive entry was in the database, but the supplier
tries to send it again, this should also disappear once some real
changes from replica 4 are replicated


but now I have on the changed master this 100... Errors

[23/Oct/2016:13:27:58 +0200] DSRetroclPlugin - delete_changerecord: could
not delete change record 396504 (rc: 32)
[23/Oct/2016:13:27:58 +0200] DSRetroclPlugin - delete_changerecord: could
not delete change record 396505 (rc: 32)
[23/Oct/2016:13:27:58 +0200] DSRetroclPlugin - delete_changerecord: could
not delete change record 396506 (rc: 32)
[23/Oct/2016:13:37:08 +0200] NSMMReplicationPlugin - replication keep
alive
entry 

Re: [Freeipa-users] Replica Problem (Errors)

[Ludwig Krispenz]

Hi,
On 10/23/2016 03:01 PM, Günther J. Niederwimmer wrote:

Hello,

I have added on my ipa (Master) Server this user and ACI with a ldif file

ldapmodify -x -D 'cn=Directory Manager' -W
dn: uid=system,cn=sysaccounts,cn=etc,dc=example,dc=com
changetype: add
objectclass: account
objectclass: simplesecurityobject
uid: system
userPassword: secret123
passwordExpirationTime: 20380119031407Z
nsIdleTimeout: 0

^D

dn: cn=users,cn=accounts,dc=example,dc=com
changetype: modify
add: aci
aci: (targetattr="mailAlternateAddress")
(targetfilter="(objectClass=mailrecipient)")
   (version
   3.0; acl "Allow system account to read mail address"; allow(read,
   search, compare) userdn =
   "ldap:///uid=system,cn=sysaccounts,cn=etc,dc=example,dc=com;;)

This Ends with a
modifying entry "cn=users,cn=accounts,dc=example,dc=com"


these changes are not related to the errors you report below (I would be 
really surprised) and you only need to apply them on one server, that's 
what replication is good for.


There are a couple of different types of messages:
- failed to delete changelog record: this is from retro changelog 
trimming, when miscalculation of the starting point for trimming starts 
with changenumber lower than what's in the retro changelog.
In my experience this can happen after a crash/kill/reboot and should 
stop after som time


- attrlist_replace errors: looks like you have recreated a replica on a 
machine and not cleaned the RUV, please see: 
http://www.freeipa.org/page/Troubleshooting#Obsolete_RUV_records


- keep-alive already exists: this is also an indication of a new 
replica, the keep alive entry was in the database, but the supplier 
tries to send it again, this should also disappear once some real 
changes from replica 4 are replicated


but now I have on the changed master this 100... Errors

[23/Oct/2016:13:27:58 +0200] DSRetroclPlugin - delete_changerecord: could not
delete change record 396504 (rc: 32)
[23/Oct/2016:13:27:58 +0200] DSRetroclPlugin - delete_changerecord: could not
delete change record 396505 (rc: 32)
[23/Oct/2016:13:27:58 +0200] DSRetroclPlugin - delete_changerecord: could not
delete change record 396506 (rc: 32)
[23/Oct/2016:13:37:08 +0200] NSMMReplicationPlugin - replication keep alive
entry 

Re: [Freeipa-users] Best and Secure Way for a System Account

[Ludwig Krispenz]

On 10/21/2016 04:05 PM, Günther J. Niederwimmer wrote:

Hello,

Thanks for the answer,

Am Freitag, 21. Oktober 2016, 07:11:58 schrieb Rich Megginson:

On 10/21/2016 06:42 AM, Günther J. Niederwimmer wrote:

Hello Martin and List,

Pardon me, but anything is wrong with the ldif i

ldapmodify -D 'cn=Directory Manager' -W -f alias.ldif
Enter LDAP Password:
ldapmodify: invalid format (line 5) entry:
"cn=users,cn=accounts,dc=4gjn,dc=com"

dn: cn=users,cn=accounts,dc=4gjn,dc=com

this is in the ldif ?

"""
dn: cn=users,cn=accounts,dc=example,dc=com
changetype: modify
add: aci
aci:
(targetattr="mailAlternateAddress")(targetfilter="(objectClass=mailrecipient)")
(version
3.0; acl "Allow system account to read mail address"; allow(read,
search, compare) userdn =
"ldap:///uid=system,cn=sysaccounts,cn=etc,dc=example,dc=com;;)
""

but what is wrong ?
the value for the aci attribute spans multiple lines.  In a ldif file a 
continuation line has to start with a space. Try


dn: cn=users,cn=accounts,dc=example,dc=com
changetype: modify
add: aci
aci: 
(targetattr="mailAlternateAddress")(targetfilter="(objectClass=mailrecipient)")
 (version
 3.0; acl "Allow system account to read mail address"; allow(read,
 search, compare) userdn =
 "ldap:///uid=system,cn=sysaccounts,cn=etc,dc=example,dc=com;;)


  

I have search and read now any Days, but this FreeIPA / LDAP Problem have
a to high level for me :-(.

Pleas help again..

Thanks for a answer

Am Montag, 17. Oktober 2016, 14:41:01 schrieb Martin Babinsky:

On 10/17/2016 02:25 PM, Günther J. Niederwimmer wrote:

Hello Martin and List

Thanks for the answer and Help.

I mean my big Problem is to understand the way to configure a ACI :-(.

# ldapmodify -x -D 'cn=Directory Manager' -W

   dn: uid=system,cn=sysaccounts,cn=etc,dc=example,dc=com
   changetype: add
   objectclass: account
   objectclass: simplesecurityobject
   uid: system
   userPassword: secret123
   passwordExpirationTime: 20380119031407Z
   nsIdleTimeout: 0
   

^D


https://www.freeipa.org/page/HowTo/LDAP#System_Accounts

The IPA Docs have no time stamp to found out, is this actual or old
:-(.

Thanks for a answer,

Hi Gunther,

that LDIF look ok to me.

Do not forget that you must set up the correct ACIs in order for the
system account to see the 'mailAlternaleAddress' attribute.

See the following document for a step-by-step guide on how to write ACIs:

https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/10
/ht
ml/Administration_Guide/Managing_Access_Control-Creating_ACIs_Manually.h
tml

To allow the system account read access to your custom attributes, you
can use LDIF like this (untested, hopefully I got it right from the top
of my head):

"""
dn: cn=users,cn=accounts,dc=example,dc=com
changetype: modify
add: aci
aci:
(targetattr="mailAlternateAddress")(targetfilter="(objectClass=mailrecipi
ent )")(version 3.0; acl "Allow system account to read mail address";
allow(read,
search, compare) userdn =
"ldap:///uid=system,cn=sysaccounts,cn=etc,dc=example,dc=com;;)
"""
save it to file and then call

ldapmodify -D 'cn=Directory Manager' -W -f aci.ldif

to add this ACI to cn=users subtree. The ACI then applies to all entries
in the subtree.


--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] replica DS failure deadlock

[Ludwig Krispenz]

On 10/19/2016 06:28 PM, Andrew E. Bruno wrote:

On Wed, Oct 19, 2016 at 05:41:37PM +0200, Ludwig Krispenz wrote:

On 10/19/2016 05:02 PM, Ludwig Krispenz wrote:

On 10/19/2016 03:48 PM, Andrew E. Bruno wrote:

On Wed, Oct 19, 2016 at 10:13:26AM +0200, Ludwig Krispenz wrote:

On 10/18/2016 08:52 PM, Andrew E. Bruno wrote:

We had one of our replicas fail today with the following errors:


[18/Oct/2016:13:40:47 -0400]
agmt="cn=meTosrv-m14-32.cbls.ccr.buffalo.edu"
(srv-m14-32:389) - Can't locate CSN 58065ef300010003 in
the changelog (DB rc=-30988). If replication stops, the
consumer may need to be reinitialized.
[18/Oct/2016:13:43:07 -0400] NSMMReplicationPlugin -
changelog program - _cl5WriteOperationTxn: retry (49) the
transaction (csn=58065f7400050004) failed (rc=-30993
(BDB0068 DB_LOCK_DEADLOCK: Locker killed to resolve a
deadlock))
[18/Oct/2016:13:43:07 -0400] NSMMReplicationPlugin -
changelog program - _cl5WriteOperationTxn: failed to write
entry with csn (58065f7400050004); db error - -30993
BDB0068 DB_LOCK_DEADLOCK: Locker killed to resolve a
deadlock
[18/Oct/2016:13:43:07 -0400] NSMMReplicationPlugin -
write_changelog_and_ruv: can't add a change for
uid=janedoe,cn=users,cn=accounts,dc=cbls,dc=ccr,dc=buffalo,dc=edu
(uniqid: 939bca48-2ced11e5-ac0b8f7e-e0b1a377, optype: 64) to
changelog csn 58065f7400050004
[18/Oct/2016:13:43:07 -0400] -
SLAPI_PLUGIN_BE_TXN_POST_MODRDN_FN plugin returned error but
did not set SLAPI_RESULT_CODE
[18/Oct/2016:13:43:07 -0400] NSMMReplicationPlugin -
process_postop: Failed to apply update
(58065f7400050004) error (1).  Aborting replication
session(conn=1314106 op=1688559)
[18/Oct/2016:13:43:12 -0400] - cos_cache_change_notify:
modified entry is NULL--updating cache just in case
[18/Oct/2016:13:43:12 -0400] - Skipping CoS Definition
cn=Password
Policy,cn=accounts,dc=cbls,dc=ccr,dc=buffalo,dc=edu--no CoS
Templates found, which should be added before the CoS
Definition.
[18/Oct/2016:13:43:20 -0400] - Operation error fetching Null
DN (4a729f9a-955a11e6-aaffa516-e778e883), error -30993.
[18/Oct/2016:13:43:20 -0400] - dn2entry_ext: Failed to get
id for changenumber=30856302,cn=changelog from entryrdn
index (-30993)
[18/Oct/2016:13:43:20 -0400] - Operation error fetching
changenumber=30856302,cn=changelog (null), error -30993.
[18/Oct/2016:13:43:20 -0400] DSRetroclPlugin - replog: an
error occured while adding change number 30856302, dn =
changenumber=30856302,cn=changelog: Operations error.
[18/Oct/2016:13:43:20 -0400] retrocl-plugin -
retrocl_postob: operation failure [1]
[18/Oct/2016:13:43:20 -0400] NSMMReplicationPlugin -
process_postop: Failed to apply update
(58065f9f0060) error (1).  Aborting replication
session(conn=1901274 op=5)
[18/Oct/2016:13:43:24 -0400] - ldbm_back_seq deadlock retry
BAD 1601, err=0 BDB0062 Successful return: 0
[18/Oct/2016:13:43:25 -0400] NSMMReplicationPlugin -
changelog program - _cl5WriteOperationTxn: retry (49) the
transaction (csn=58065f7c000a0004) failed (rc=-30993
(BDB0068 DB_LOCK_DEADLOCK: Locker killed to resolve a
deadlock))
[18/Oct/2016:13:43:25 -0400] NSMMReplicationPlugin -
changelog program - _cl5WriteOperationTxn: failed to write
entry with csn (58065f7c000a0004); db error - -30993
BDB0068 DB_LOCK_DEADLOCK: Locker killed to resolve a
deadlock
[18/Oct/2016:13:43:25 -0400] NSMMReplicationPlugin -
write_changelog_and_ruv: can't add a change for
uid=janedoe,cn=users,cn=accounts,dc=cbls,dc=ccr,dc=buffalo,dc=edu
(uniqid: 4080421a-2d0211e5-ac0b8f7e-e0b1a377, optype: 64) to
changelog csn 58065f7c000a0004


ns-slapd was hung so we restarted and now it's stuck and
won't come back up. It
hangs up here:

[18/Oct/2016:14:12:31 -0400] - Skipping CoS Definition
cn=Password
Policy,cn=accounts,dc=cbls,dc=ccr,dc=buffalo,dc=edu--no CoS
Templates found, which should be added before the CoS
Definition.
[18/Oct/2016:14:12:31 -0400] NSMMReplicationPlugin -
changelog program - _cl5NewDBFile: PR_DeleteSemaphore: 
/var/lib/dirsrv/slapd-CBLS-CCR-BUFFALO-EDU/cldb/a32992ce-71b811e5-9d33a516-e778e883.sema;
NSPR error - -5943
[18/Oct/2016:14:12:32 -0400] NSMMReplicationPlugin -
changelog program - _cl5NewDBFile: PR_DeleteSemaphore: 
/var/lib/dirsrv/slapd-CBLS-CCR-BUFFALO-EDU/cldb/986efe12-71b811e5-9d33a516-e778e883.sema;
NSPR error - -5943


Tried deleting the semaphore files and restarting but no
luck. Attached
is a stacktrace of the stuck ns-slapd process.

Here's the versions were running:

ipa-server-4.2.0-15.0.1.el7.centos.19.x86_64
389-ds-base-1.3.4.0-33.el7_2.x86_64

FWIW, we were experimenting with the new life-cycle
management features,
specifically "preserved" users and deleted the user
"janedoe" when this
happened.  From the errors above looks like this host failed to
replicate the change?  Not sure if this is related or not.

Is it possible to recover the database? Thanks in advance
for any pointers.

from the stack trace the process is not hanging, it is trying to
recover.
After a 

Re: [Freeipa-users] replica DS failure deadlock

[Ludwig Krispenz]

On 10/19/2016 05:02 PM, Ludwig Krispenz wrote:


On 10/19/2016 03:48 PM, Andrew E. Bruno wrote:

On Wed, Oct 19, 2016 at 10:13:26AM +0200, Ludwig Krispenz wrote:

On 10/18/2016 08:52 PM, Andrew E. Bruno wrote:

We had one of our replicas fail today with the following errors:


[18/Oct/2016:13:40:47 -0400] 
agmt="cn=meTosrv-m14-32.cbls.ccr.buffalo.edu" (srv-m14-32:389) - 
Can't locate CSN 58065ef300010003 in the changelog (DB 
rc=-30988). If replication stops, the consumer may need to be 
reinitialized.
[18/Oct/2016:13:43:07 -0400] NSMMReplicationPlugin - changelog 
program - _cl5WriteOperationTxn: retry (49) the transaction 
(csn=58065f7400050004) failed (rc=-30993 (BDB0068 
DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock))
[18/Oct/2016:13:43:07 -0400] NSMMReplicationPlugin - changelog 
program - _cl5WriteOperationTxn: failed to write entry with csn 
(58065f7400050004); db error - -30993 BDB0068 DB_LOCK_DEADLOCK: 
Locker killed to resolve a deadlock
[18/Oct/2016:13:43:07 -0400] NSMMReplicationPlugin - 
write_changelog_and_ruv: can't add a change for 
uid=janedoe,cn=users,cn=accounts,dc=cbls,dc=ccr,dc=buffalo,dc=edu 
(uniqid: 939bca48-2ced11e5-ac0b8f7e-e0b1a377, optype: 64) to 
changelog csn 58065f7400050004
[18/Oct/2016:13:43:07 -0400] - SLAPI_PLUGIN_BE_TXN_POST_MODRDN_FN 
plugin returned error but did not set SLAPI_RESULT_CODE
[18/Oct/2016:13:43:07 -0400] NSMMReplicationPlugin - 
process_postop: Failed to apply update (58065f7400050004) error 
(1).  Aborting replication session(conn=1314106 op=1688559)
[18/Oct/2016:13:43:12 -0400] - cos_cache_change_notify: modified 
entry is NULL--updating cache just in case
[18/Oct/2016:13:43:12 -0400] - Skipping CoS Definition cn=Password 
Policy,cn=accounts,dc=cbls,dc=ccr,dc=buffalo,dc=edu--no CoS 
Templates found, which should be added before the CoS Definition.
[18/Oct/2016:13:43:20 -0400] - Operation error fetching Null DN 
(4a729f9a-955a11e6-aaffa516-e778e883), error -30993.
[18/Oct/2016:13:43:20 -0400] - dn2entry_ext: Failed to get id for 
changenumber=30856302,cn=changelog from entryrdn index (-30993)
[18/Oct/2016:13:43:20 -0400] - Operation error fetching 
changenumber=30856302,cn=changelog (null), error -30993.
[18/Oct/2016:13:43:20 -0400] DSRetroclPlugin - replog: an error 
occured while adding change number 30856302, dn = 
changenumber=30856302,cn=changelog: Operations error.
[18/Oct/2016:13:43:20 -0400] retrocl-plugin - retrocl_postob: 
operation failure [1]
[18/Oct/2016:13:43:20 -0400] NSMMReplicationPlugin - 
process_postop: Failed to apply update (58065f9f0060) error 
(1).  Aborting replication session(conn=1901274 op=5)
[18/Oct/2016:13:43:24 -0400] - ldbm_back_seq deadlock retry BAD 
1601, err=0 BDB0062 Successful return: 0
[18/Oct/2016:13:43:25 -0400] NSMMReplicationPlugin - changelog 
program - _cl5WriteOperationTxn: retry (49) the transaction 
(csn=58065f7c000a0004) failed (rc=-30993 (BDB0068 
DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock))
[18/Oct/2016:13:43:25 -0400] NSMMReplicationPlugin - changelog 
program - _cl5WriteOperationTxn: failed to write entry with csn 
(58065f7c000a0004); db error - -30993 BDB0068 DB_LOCK_DEADLOCK: 
Locker killed to resolve a deadlock
[18/Oct/2016:13:43:25 -0400] NSMMReplicationPlugin - 
write_changelog_and_ruv: can't add a change for 
uid=janedoe,cn=users,cn=accounts,dc=cbls,dc=ccr,dc=buffalo,dc=edu 
(uniqid: 4080421a-2d0211e5-ac0b8f7e-e0b1a377, optype: 64) to 
changelog csn 58065f7c000a0004



ns-slapd was hung so we restarted and now it's stuck and won't come 
back up. It

hangs up here:

[18/Oct/2016:14:12:31 -0400] - Skipping CoS Definition cn=Password 
Policy,cn=accounts,dc=cbls,dc=ccr,dc=buffalo,dc=edu--no CoS 
Templates found, which should be added before the CoS Definition.
[18/Oct/2016:14:12:31 -0400] NSMMReplicationPlugin - changelog 
program - _cl5NewDBFile: PR_DeleteSemaphore: 
/var/lib/dirsrv/slapd-CBLS-CCR-BUFFALO-EDU/cldb/a32992ce-71b811e5-9d33a516-e778e883.sema; 
NSPR error - -5943
[18/Oct/2016:14:12:32 -0400] NSMMReplicationPlugin - changelog 
program - _cl5NewDBFile: PR_DeleteSemaphore: 
/var/lib/dirsrv/slapd-CBLS-CCR-BUFFALO-EDU/cldb/986efe12-71b811e5-9d33a516-e778e883.sema; 
NSPR error - -5943



Tried deleting the semaphore files and restarting but no luck. 
Attached

is a stacktrace of the stuck ns-slapd process.

Here's the versions were running:

ipa-server-4.2.0-15.0.1.el7.centos.19.x86_64
389-ds-base-1.3.4.0-33.el7_2.x86_64

FWIW, we were experimenting with the new life-cycle management 
features,
specifically "preserved" users and deleted the user "janedoe" when 
this

happened.  From the errors above looks like this host failed to
replicate the change?  Not sure if this is related or not.

Is it possible to recover the database? Thanks in advance for any 
pointers.
from the stack trace the process is not hanging, it is trying to 
recover.

After a crash/kill  the changelog does not contai a RUV and it i

Re: [Freeipa-users] replica DS failure deadlock

[Ludwig Krispenz]

On 10/19/2016 03:48 PM, Andrew E. Bruno wrote:

On Wed, Oct 19, 2016 at 10:13:26AM +0200, Ludwig Krispenz wrote:

On 10/18/2016 08:52 PM, Andrew E. Bruno wrote:

We had one of our replicas fail today with the following errors:


[18/Oct/2016:13:40:47 -0400] agmt="cn=meTosrv-m14-32.cbls.ccr.buffalo.edu" 
(srv-m14-32:389) - Can't locate CSN 58065ef300010003 in the changelog (DB rc=-30988). 
If replication stops, the consumer may need to be reinitialized.
[18/Oct/2016:13:43:07 -0400] NSMMReplicationPlugin - changelog program - 
_cl5WriteOperationTxn: retry (49) the transaction (csn=58065f7400050004) 
failed (rc=-30993 (BDB0068 DB_LOCK_DEADLOCK: Locker killed to resolve a 
deadlock))
[18/Oct/2016:13:43:07 -0400] NSMMReplicationPlugin - changelog program - 
_cl5WriteOperationTxn: failed to write entry with csn (58065f7400050004); 
db error - -30993 BDB0068 DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock
[18/Oct/2016:13:43:07 -0400] NSMMReplicationPlugin - write_changelog_and_ruv: 
can't add a change for 
uid=janedoe,cn=users,cn=accounts,dc=cbls,dc=ccr,dc=buffalo,dc=edu (uniqid: 
939bca48-2ced11e5-ac0b8f7e-e0b1a377, optype: 64) to changelog csn 
58065f7400050004
[18/Oct/2016:13:43:07 -0400] - SLAPI_PLUGIN_BE_TXN_POST_MODRDN_FN plugin 
returned error but did not set SLAPI_RESULT_CODE
[18/Oct/2016:13:43:07 -0400] NSMMReplicationPlugin - process_postop: Failed to 
apply update (58065f7400050004) error (1).  Aborting replication 
session(conn=1314106 op=1688559)
[18/Oct/2016:13:43:12 -0400] - cos_cache_change_notify: modified entry is 
NULL--updating cache just in case
[18/Oct/2016:13:43:12 -0400] - Skipping CoS Definition cn=Password 
Policy,cn=accounts,dc=cbls,dc=ccr,dc=buffalo,dc=edu--no CoS Templates found, 
which should be added before the CoS Definition.
[18/Oct/2016:13:43:20 -0400] - Operation error fetching Null DN 
(4a729f9a-955a11e6-aaffa516-e778e883), error -30993.
[18/Oct/2016:13:43:20 -0400] - dn2entry_ext: Failed to get id for 
changenumber=30856302,cn=changelog from entryrdn index (-30993)
[18/Oct/2016:13:43:20 -0400] - Operation error fetching 
changenumber=30856302,cn=changelog (null), error -30993.
[18/Oct/2016:13:43:20 -0400] DSRetroclPlugin - replog: an error occured while 
adding change number 30856302, dn = changenumber=30856302,cn=changelog: 
Operations error.
[18/Oct/2016:13:43:20 -0400] retrocl-plugin - retrocl_postob: operation failure 
[1]
[18/Oct/2016:13:43:20 -0400] NSMMReplicationPlugin - process_postop: Failed to 
apply update (58065f9f0060) error (1).  Aborting replication 
session(conn=1901274 op=5)
[18/Oct/2016:13:43:24 -0400] - ldbm_back_seq deadlock retry BAD 1601, err=0 
BDB0062 Successful return: 0
[18/Oct/2016:13:43:25 -0400] NSMMReplicationPlugin - changelog program - 
_cl5WriteOperationTxn: retry (49) the transaction (csn=58065f7c000a0004) 
failed (rc=-30993 (BDB0068 DB_LOCK_DEADLOCK: Locker killed to resolve a 
deadlock))
[18/Oct/2016:13:43:25 -0400] NSMMReplicationPlugin - changelog program - 
_cl5WriteOperationTxn: failed to write entry with csn (58065f7c000a0004); 
db error - -30993 BDB0068 DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock
[18/Oct/2016:13:43:25 -0400] NSMMReplicationPlugin - write_changelog_and_ruv: 
can't add a change for 
uid=janedoe,cn=users,cn=accounts,dc=cbls,dc=ccr,dc=buffalo,dc=edu (uniqid: 
4080421a-2d0211e5-ac0b8f7e-e0b1a377, optype: 64) to changelog csn 
58065f7c000a0004


ns-slapd was hung so we restarted and now it's stuck and won't come back up. It
hangs up here:

[18/Oct/2016:14:12:31 -0400] - Skipping CoS Definition cn=Password 
Policy,cn=accounts,dc=cbls,dc=ccr,dc=buffalo,dc=edu--no CoS Templates found, 
which should be added before the CoS Definition.
[18/Oct/2016:14:12:31 -0400] NSMMReplicationPlugin - changelog program - 
_cl5NewDBFile: PR_DeleteSemaphore: 
/var/lib/dirsrv/slapd-CBLS-CCR-BUFFALO-EDU/cldb/a32992ce-71b811e5-9d33a516-e778e883.sema;
 NSPR error - -5943
[18/Oct/2016:14:12:32 -0400] NSMMReplicationPlugin - changelog program - 
_cl5NewDBFile: PR_DeleteSemaphore: 
/var/lib/dirsrv/slapd-CBLS-CCR-BUFFALO-EDU/cldb/986efe12-71b811e5-9d33a516-e778e883.sema;
 NSPR error - -5943


Tried deleting the semaphore files and restarting but no luck. Attached
is a stacktrace of the stuck ns-slapd process.

Here's the versions were running:

ipa-server-4.2.0-15.0.1.el7.centos.19.x86_64
389-ds-base-1.3.4.0-33.el7_2.x86_64

FWIW, we were experimenting with the new life-cycle management features,
specifically "preserved" users and deleted the user "janedoe" when this
happened.  From the errors above looks like this host failed to
replicate the change?  Not sure if this is related or not.

Is it possible to recover the database? Thanks in advance for any pointers.

from the stack trace the process is not hanging, it is trying to recover.
After a crash/kill  the changelog does not contai a RUV and it is
reconstructed by reading all records in the changelog, if this 

Re: [Freeipa-users] Lots of error messages in logs after upgrade

[Ludwig Krispenz]

On 10/19/2016 09:39 AM, Prashant Bapat wrote:

Some more info.

This is happening on one of the hosts for which replica-info file was 
generated but for some reason the replica installation failed. So I 
went ahead and deleted and created the replica file again and this 
time installation went thru fine. Should this cause logs like this ?
you now have two replicaids with the same url, you need to do a cleanruv 
as discussed frequently on this list


These messages are seen every 5 mins.

On 18 October 2016 at 22:38, Prashant Bapat > wrote:


Hi,

I'm seeing lots of error messages like this in the DS logs.

[18/Oct/2016:17:00:37 +] attrlist_replace - attr_replace
(nsslapd-referral,
ldap://ipa-primary.example.net:389/dc%3Dexample%2Cdc%3Dnet
) failed.
[18/Oct/2016:17:00:37 +] attrlist_replace - attr_replace
(nsslapd-referral,
ldap://ipa-primary.example.net:389/dc%3Dexample%2Cdc%3Dnet
) failed.
[18/Oct/2016:17:00:37 +] attrlist_replace - attr_replace
(nsslapd-referral,
ldap://ipa-primary.example.net:389/dc%3Dexample%2Cdc%3Dnet
) failed.
[18/Oct/2016:17:00:46 +] attrlist_replace - attr_replace
(nsslapd-referral,
ldap://ipa-primary.example.net:389/dc%3Dexample%2Cdc%3Dnet
) failed.
[18/Oct/2016:17:00:46 +] attrlist_replace - attr_replace
(nsslapd-referral,
ldap://ipa-primary.example.net:389/dc%3Dexample%2Cdc%3Dnet
) failed.
[18/Oct/2016:17:00:46 +] attrlist_replace - attr_replace
(nsslapd-referral,
ldap://ipa-primary.example.net:389/dc%3Dexample%2Cdc%3Dnet
) failed.
[18/Oct/2016:17:00:46 +] attrlist_replace - attr_replace
(nsslapd-referral,
ldap://ipa-primary.example.net:389/dc%3Dexample%2Cdc%3Dnet
) failed.
[18/Oct/2016:17:00:46 +] attrlist_replace - attr_replace
(nsslapd-referral,
ldap://ipa-primary.example.net:389/dc%3Dexample%2Cdc%3Dnet
) failed.
[18/Oct/2016:17:00:46 +] attrlist_replace - attr_replace
(nsslapd-referral,
ldap://ipa-primary.example.net:389/dc%3Dexample%2Cdc%3Dnet
) failed.

We moved from 4.1.4 (FC21) to 4.2.0 (Centos7.2) recently. We have
total 8 IPA servers with replication. Below are the steps I followed.

1. Install a new Centos server.
2. Replicated against a Fedora server with CA.
3. Moved the DNA ranges.
4. From the Centos master created replicas.

Is this related to the DS package version ? We
have 389-ds-base-1.3.4.0-33.el7_2.x86_64.

Thanks.
--Prashant






--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] replica DS failure deadlock

[Ludwig Krispenz]

On 10/18/2016 08:52 PM, Andrew E. Bruno wrote:

We had one of our replicas fail today with the following errors:


[18/Oct/2016:13:40:47 -0400] agmt="cn=meTosrv-m14-32.cbls.ccr.buffalo.edu" 
(srv-m14-32:389) - Can't locate CSN 58065ef300010003 in the changelog (DB rc=-30988). 
If replication stops, the consumer may need to be reinitialized.
[18/Oct/2016:13:43:07 -0400] NSMMReplicationPlugin - changelog program - 
_cl5WriteOperationTxn: retry (49) the transaction (csn=58065f7400050004) 
failed (rc=-30993 (BDB0068 DB_LOCK_DEADLOCK: Locker killed to resolve a 
deadlock))
[18/Oct/2016:13:43:07 -0400] NSMMReplicationPlugin - changelog program - 
_cl5WriteOperationTxn: failed to write entry with csn (58065f7400050004); 
db error - -30993 BDB0068 DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock
[18/Oct/2016:13:43:07 -0400] NSMMReplicationPlugin - write_changelog_and_ruv: 
can't add a change for 
uid=janedoe,cn=users,cn=accounts,dc=cbls,dc=ccr,dc=buffalo,dc=edu (uniqid: 
939bca48-2ced11e5-ac0b8f7e-e0b1a377, optype: 64) to changelog csn 
58065f7400050004
[18/Oct/2016:13:43:07 -0400] - SLAPI_PLUGIN_BE_TXN_POST_MODRDN_FN plugin 
returned error but did not set SLAPI_RESULT_CODE
[18/Oct/2016:13:43:07 -0400] NSMMReplicationPlugin - process_postop: Failed to 
apply update (58065f7400050004) error (1).  Aborting replication 
session(conn=1314106 op=1688559)
[18/Oct/2016:13:43:12 -0400] - cos_cache_change_notify: modified entry is 
NULL--updating cache just in case
[18/Oct/2016:13:43:12 -0400] - Skipping CoS Definition cn=Password 
Policy,cn=accounts,dc=cbls,dc=ccr,dc=buffalo,dc=edu--no CoS Templates found, 
which should be added before the CoS Definition.
[18/Oct/2016:13:43:20 -0400] - Operation error fetching Null DN 
(4a729f9a-955a11e6-aaffa516-e778e883), error -30993.
[18/Oct/2016:13:43:20 -0400] - dn2entry_ext: Failed to get id for 
changenumber=30856302,cn=changelog from entryrdn index (-30993)
[18/Oct/2016:13:43:20 -0400] - Operation error fetching 
changenumber=30856302,cn=changelog (null), error -30993.
[18/Oct/2016:13:43:20 -0400] DSRetroclPlugin - replog: an error occured while 
adding change number 30856302, dn = changenumber=30856302,cn=changelog: 
Operations error.
[18/Oct/2016:13:43:20 -0400] retrocl-plugin - retrocl_postob: operation failure 
[1]
[18/Oct/2016:13:43:20 -0400] NSMMReplicationPlugin - process_postop: Failed to 
apply update (58065f9f0060) error (1).  Aborting replication 
session(conn=1901274 op=5)
[18/Oct/2016:13:43:24 -0400] - ldbm_back_seq deadlock retry BAD 1601, err=0 
BDB0062 Successful return: 0
[18/Oct/2016:13:43:25 -0400] NSMMReplicationPlugin - changelog program - 
_cl5WriteOperationTxn: retry (49) the transaction (csn=58065f7c000a0004) 
failed (rc=-30993 (BDB0068 DB_LOCK_DEADLOCK: Locker killed to resolve a 
deadlock))
[18/Oct/2016:13:43:25 -0400] NSMMReplicationPlugin - changelog program - 
_cl5WriteOperationTxn: failed to write entry with csn (58065f7c000a0004); 
db error - -30993 BDB0068 DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock
[18/Oct/2016:13:43:25 -0400] NSMMReplicationPlugin - write_changelog_and_ruv: 
can't add a change for 
uid=janedoe,cn=users,cn=accounts,dc=cbls,dc=ccr,dc=buffalo,dc=edu (uniqid: 
4080421a-2d0211e5-ac0b8f7e-e0b1a377, optype: 64) to changelog csn 
58065f7c000a0004


ns-slapd was hung so we restarted and now it's stuck and won't come back up. It
hangs up here:

[18/Oct/2016:14:12:31 -0400] - Skipping CoS Definition cn=Password 
Policy,cn=accounts,dc=cbls,dc=ccr,dc=buffalo,dc=edu--no CoS Templates found, 
which should be added before the CoS Definition.
[18/Oct/2016:14:12:31 -0400] NSMMReplicationPlugin - changelog program - 
_cl5NewDBFile: PR_DeleteSemaphore: 
/var/lib/dirsrv/slapd-CBLS-CCR-BUFFALO-EDU/cldb/a32992ce-71b811e5-9d33a516-e778e883.sema;
 NSPR error - -5943
[18/Oct/2016:14:12:32 -0400] NSMMReplicationPlugin - changelog program - 
_cl5NewDBFile: PR_DeleteSemaphore: 
/var/lib/dirsrv/slapd-CBLS-CCR-BUFFALO-EDU/cldb/986efe12-71b811e5-9d33a516-e778e883.sema;
 NSPR error - -5943


Tried deleting the semaphore files and restarting but no luck. Attached
is a stacktrace of the stuck ns-slapd process.

Here's the versions were running:

ipa-server-4.2.0-15.0.1.el7.centos.19.x86_64
389-ds-base-1.3.4.0-33.el7_2.x86_64

FWIW, we were experimenting with the new life-cycle management features,
specifically "preserved" users and deleted the user "janedoe" when this
happened.  From the errors above looks like this host failed to
replicate the change?  Not sure if this is related or not.

Is it possible to recover the database? Thanks in advance for any pointers.
from the stack trace the process is not hanging, it is trying to 
recover. After a crash/kill  the changelog does not contai a RUV and it 
is reconstructed by reading all records in the changelog, if this is 
large it can take some time.

If you look at that part of the stack repeatedly,

#4  0x7f4e88daeba5 in cl5DBData2Entry (data=, len=, 

Re: [Freeipa-users] Different Database Generation ID

[Ludwig Krispenz]

Hi,

you get the "different database generation" if one side is built from 
scratch or reimported from a plain ldif without repl stat e information. 
replication will only work if both sides have the same data origin.


About initlializing back and forth it depends on your topology if it can 
become a problem. If a replica is reinitialized it's changelog is 
recreated (the old one will no longer match) and if you do it again in 
the other direction you remove the changelog there as well - and then 
can msis changes not yet replicated to other replicas and you can run 
into the "csn not found problems".


I looked up one of your previous posts about which version of 389-ds you 
are using, and it looks like you have one we know has some issues, as 
stated several times on this list :-(


About your observation that replication is stopping and working again 
after restarting, this can be a problem of the replication agreement 
going into fatal state instead of retrying. Restarting the server 
overcomes this, but you could achieve it by disabling the agreement.


Ludwig

On 10/11/2016 06:13 PM, Ian Harding wrote:

I have this error in the log of my FreeIPA server freeipa-sea.bpt.rocks:

[11/Oct/2016:09:04:39 -0700] NSMMReplicationPlugin -
agmt="cn=masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat"
(seattlenfs:389): The remote replica has a different database generation
ID than the local database.  You may have to reinitialize the remote
replica, or the local replica.

So I did this:

ipa-replica-manage re-initialize --from freeipa-sea.bpt.rocks

on seattlenfs

But the error continues.

I think I know why.  freeipa-sea had a meltdown and I had to rebuild it,
and established it as a replica of seattlenfs.  Unfortunately, I think
seattlenfs was a replica of the original freeipa-sea.

It seems like a bad idea to reinitialize themselves from each other, and
in fact it's warned against here:

https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/8.2/html/Administration_Guide/Managing_Replication-Troubleshooting_Replication_Related_Problems.html

"... Also, M2 should not initialize M1 back."

But in looking at my bash history I have indeed done that as well.

Is there any way out of this mess?  These two servers actually DO
replicate, most of the time.  They stop for no reason and restarting the
ipa services on freeipa-sea does get them started again.



--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Replication attrlist_replace nsslapd-referral failed

[Ludwig Krispenz]

Hi,

you don't specify the version you are using:
If it is 389-ds-base-1.3.4.0-33.el7_2.x86_64
the following may apply:
>>>
we have identified an issue with this version, it includes a fix for 
389-ds ticket #48766, which was incomplete and resolved shortly after 
the release of this version (it is missing the latest patch for #49766 
and for #48954).
You can try to go back to 1.3.4.0-32 or if you have support get a hotfix 
from our support.

<<<

Sorry for this,
On 10/11/2016 03:48 AM, Fil Di Noto wrote:

After an IPA server is re-initialized it immediately begins failing
incremental updates. I checked the kerberos logs and things appear to
be ok there, I can manually test LDAP from all servers against all
other servers.

There is an DS5ReplicaBindDN entry in "dn:
cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config" for
an IPA server that no longer exists. But all IPA living servers have
an entry for all other living servers.
There is the correct number of cn=master, and cn=ca, and the
caRenewalMaster is set on the correct master.

  "ipa-replica-manage del --force --clean " does not remove the entry.

There were some RUV from the old servers also and I cleaned them. The
man page says if a clean is run on the wrong ID then the server should
be re-initialized, so I just did that on purpose and re-initialized
the one of the servers and that has cleared the NSMMReplicationPlugin
error (so far) but I am still getting the attrlist_replace error.

I'm getting no indication of kerberos problems.Could it be the
NSACLPlugin ? It preceeds the other error every time but that is
probably just regular startup procedure, and having an ACL for
something that doesn't exist doesn't feel like a fatal error to me. I
didn't do the KRA install.

[root@ipa05 slapd-example-com]# tail -f errors
[10/Oct/2016:23:27:57 +] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=example,dc=com does not exist
[10/Oct/2016:23:27:57 +] NSACLPlugin - The ACL target
cn=casigningcert
cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=example,dc=com does not
exist
[10/Oct/2016:23:27:57 +] agmt="cn=meToipa07.example.com"
(ipa07:389) - Can't locate CSN 57fc2e7f000a000d in the changelog
(DB rc=-30988). If replication stops, the consumer may need to be
reinitialized.
[10/Oct/2016:23:27:57 +] NSMMReplicationPlugin - changelog program
- agmt="cn=meToipa07.example.com" (ipa07:389): CSN
57fc2e7f000a000d not found, we aren't as up to date, or we purged
[10/Oct/2016:23:27:57 +] NSMMReplicationPlugin -
agmt="cn=meToipa07.example.com" (ipa07:389): Data required to update
replica has been purged. The replica must be reinitialized.
[10/Oct/2016:23:27:57 +] NSMMReplicationPlugin -
agmt="cn=meToipa07.example.com" (ipa07:389): Incremental update failed
and requires administrator action
[10/Oct/2016:23:29:09 +] attrlist_replace - attr_replace
(nsslapd-referral, ldap://ipa07.example.com:389/o%3Dipaca) failed.



--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Question about removed replica, take two

[Ludwig Krispenz]

Hi,

the RUV in the replication agreement is maintained to control changelog 
trimming, no changes should be deleted from the changelog which have not 
been seen by all consumers. Since not always a connection for a 
replication agreement can be established, eg if the consumer is down, 
this information is made persistent and kept in the replication agreement.
So, if you have references to removed servers in the agreement this 
should do no harm since teh changes have alredy be removed from the 
changelog during cleanallruv.
The only scenario a problem could arise is if you reinstall a replica on 
one of the removed with a new replica ID, then you could end up with two 
replica ids with the same url and get the attrlist_replace errors.


The removal of the replica id from the replication agreement RUV is noe 
handled by cleanallruv (upstream ticket #48414), but you can edit the 
dse.ldif and remove them manually


Regards,
Ludwig

On 10/05/2016 03:07 PM, John Desantis wrote:

Hello all (again),

I think my reference to a disease prevented my message from being
delivered, despite seeing it posted on the list archive.  I apologize
in advance for the additional "noise".

Anyways, I was hoping some lingering questions could be answered
regarding some visible entries via ldapsearch, which manifest a
removed replica's hostname [1].

Running the ipa-replica-manage and ipa-csreplica-manage commands do
not show the host in question any longer, but when I run a few
directory searches on each replica using the commands below:

# ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
manager" -b "cn=config" objectclass=nsds5replica
# ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
manager" -b "cn=config" objectclass=nsds5replicationagreement

I'm able to see the old host on the master, but not on the replicas.  See below.

# master, replica id 4:
ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
manager" -b "cn=config" objectclass=nsds5replica|grep oldhost
nsDS5ReplicaBindDN:
krbprincipalname=ldap/oldhost.dom.dom@dom.dom.dom,cn=services,cn=accounts,dc=dom,dc=dom,dc=dom

ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
manager" -b "cn=config" objectclass=nsds5replicationagreement|grep
oldhost
nsds50ruv: {replica 24 ldap://oldhost.dom.dom.dom:389}
5447f2520018 5447f8610018
nsruvReplicaLastModified: {replica 24 ldap://oldhost.dom.dom.dom:389} 
nsds50ruv: {replica 24 ldap://oldhost.dom.dom.dom:389}
5447f2520018 5447f56b00020018
nsruvReplicaLastModified: {replica 24 ldap://oldhost.dom.dom.dom:389} 

It's listed twice due to the other hosts in the topology.

# replica id 22
ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
manager" -b "cn=config" objectclass=nsds5replica|grep oldhost
ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
manager" -b "cn=config" objectclass=nsds5replicationagreement|grep
oldhost

# replica id 21
ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
manager" -b "cn=config" objectclass=nsds5replica|grep oldhost
ldapsearch -Y GSSAPI -o ldif-wrap=no -h localhost -D "cn=directory
manager" -b "cn=config" objectclass=nsds5replicationagreement|grep
oldhost

The other two replicas no longer have the reference to the old host
after the CLEANALLRUV and CLEANRUV tasks performed by ldapmodify.  I
then read via [2] that the dse.ldif could be manually edited to remove
references, but I'm not sure if that should be done if the general
opinion is that the old references aren't going to cause a problem.
Based upon the information above, is having a reference to the hold
host via the ldapsearch outputs above going to be a problem?  If the
entry shouldn't be there, should the ldapmodify be performed against
the 
"cn=meTomaster.dom.dom.dom,cn=replica,cn=dc\3Ddom\2Cdc\3Ddom\2Cdc\3Ddom,cn=mapping
tree,cn=config" bases?

For reference, these are the commands I ran to get to state [1]:

# master
ldapmodify -x -W -h localhost -D "cn=directory manager" 

Re: [Freeipa-users] Server replication stopped working

[Ludwig Krispenz]

On 09/27/2016 06:04 PM, Youenn PIOLET wrote:

Hi Ludwig,

Version:
389-ds-base-1.3.4.0-33.el7_2.x86_64
we have identified an issue with this version, it includes a fix for 
389-ds ticket #48766, which was incomplete and resolved shortly after 
the release of this version (it is missing the latest patch for #49766 
and for #48954).
You can try to go back to 1.3.4.0-32 or if you have support get a hotfix 
from our support.


Sorry for this,
Ludwig


The timestamp probably matches the last time I've done a 
ipa-replica-manage re-initialize.
I have to do it every day (many times a day actually!), as replication 
is broken, This CSN changes all the time.


My main goal is to rebuilt everything from a clean base.
I've got no master without errors.

What is the easiest way to rebuilt everything?
ipa-[cs]replica-manage re-initialize isn't very effective.

Thanks by advance,
Regards

--
Youenn Piolet
piole...@gmail.com <mailto:piole...@gmail.com>
/
/

2016-09-26 9:42 GMT+02:00 Ludwig Krispenz <lkris...@redhat.com 
<mailto:lkris...@redhat.com>>:



On 09/25/2016 09:35 PM, Youenn PIOLET wrote:

Hi there,

Same issue for me in a my 15 ipa-servers multi-master grid just
after the update.
The replication is completely broken except on 3/15 nodes.

This is the second time I have to fully reinitialize the whole
cluster for similar reason. I don't know what to do to clean this
mess...
For more information: this cluster has been initialized on a
fedora 4.1.4 more than one year ago then complemetely migrated to
Centos 7, IPA 4.2.

what is the exact version of 389-ds-base you are running ?

did these errors come out of the blue or are they related to some
activities ? The csn which is not found has a timestamp of "Thu,
22 Sep 2016 15:59:08 GMT" did anything happen around this time ?



Example on fr-master03 error logs:

[25/Sep/2016:19:27:31 +] NSMMReplicationPlugin - changelog
program - agmt="cn=meTofr-master01.domain" (fr-master01:389): CSN
57e3ffcc0003001a not found, we aren't as up to date, or we purged
[25/Sep/2016:19:27:31 +] NSMMReplicationPlugin -
agmt="cn=meTofr-master01.domain" (fr-master01:389): Data required
to update replica has been purged. The replica must be reinitialized.
[25/Sep/2016:19:27:31 +] NSMMReplicationPlugin -
agmt="cn=meTofr-master01.domain" (fr-master01:389): Incremental
update failed and requires administrator action
ipa: INFO: The ipactl command was successful
[25/Sep/2016:19:27:35 +] agmt="cn=meTofr-master02.domain"
(fr-master02:389) - Can't locate CSN 57e3ffcc0003001a in the
changelog (DB rc=-30988). If replication stops, the consumer may
need to be reinitialized.
[25/Sep/2016:19:27:35 +] NSMMReplicationPlugin - changelog
program - agmt="cn=meTofr-master02.domain" (fr-master02:389): CSN
57e3ffcc0003001a not found, we aren't as up to date, or we purged
[25/Sep/2016:19:27:35 +] NSMMReplicationPlugin -
agmt="cn=meTofr-master02.domain" (fr-master02:389): Data required
to update replica has been purged. The replica must be reinitialized.
[25/Sep/2016:19:27:35 +] NSMMReplicationPlugin -
agmt="cn=meTofr-master02.domain" (fr-master02:389): Incremental
update failed and requires administrator action

Regards,

--
Youenn Piolet
piole...@gmail.com <mailto:piole...@gmail.com>
/
/

2016-09-23 17:51 GMT+02:00 Mike Driscoll
<mike.drisc...@oracle.com <mailto:mike.drisc...@oracle.com>>:

Hello.  I have four IPA servers replicating in full mesh. 
All four servers are running

ipa-server-4.2.0-15.0.1.el7_2.19.x86_64.

This was working for some time but now I see that no
replication is occurring automatically at present.

When I update a user attribute on an IPA server, I see errors
like these:
[22/Sep/2016:16:53:49 -0700] attrlist_replace - attr_replace
(nsslapd-referral, ldap://ldap03.xx.com:389/o%3Dipaca) failed.
[22/Sep/2016:16:58:56 -0700] NSMMReplicationPlugin -
agmt="cn=masterAgreement1-ldap03.xx.com
<http://masteragreement1-ldap03.xx.com>-pki-tomcat" (ldap03:389):
Incremental update failed and requires administrator action

I can reinitialize without errors.
ipa-csreplica-manage re-initialize --from=ldap01.xx.com
<http://ldap04.us.oracle.com>
ipa-replica-manage re-initialize --from=ldap01.xx.com
<http://ldap01.xx.com>
Afterwards I see my attribute (and other) changes are
replicated on each server I re-initialize from.  But
subsequently, replication doesn’t seem to be happening.

I reinitialized according to the steps in Table 8.7,
“Replication Error

Re: [Freeipa-users] replicas removed, but incorrectly

[Ludwig Krispenz]

you should only remove agreements to no longer existing servers, eg where:

nsDS5ReplicaHost: kdc01.unix.iriszorg.nl

the other one should remain, not sure why it cannot contact the server

On 09/26/2016 03:35 PM, Natxo Asenjo wrote:

hi,

or do I need to remove:

dn: 
cn=cloneAgreement1-kdc03.unix.iriszorg.nl-pki-tomcat,cn=replica,cn=o\3Dipa

 ca,cn=mapping tree,cn=config

because it has this:
nsds5replicaLastUpdateStatus: -1 Unable to acquire replicaLDAP error: 
Can't co

 ntact LDAP server
nsds5replicaUpdateInProgress: FALSE

and this:

dn: 
cn=masterAgreement1-kdc04.unix.iriszorg.nl-pki-tomcat,cn=replica,cn=o\3Dip

 aca,cn=mapping tree,cn=config

nsds5replicaLastUpdateStatus: -1 Incremental update has failed and 
requires ad

 ministrator actionLDAP error: Can't contact LDAP server



On Mon, Sep 26, 2016 at 3:32 PM, Natxo Asenjo <natxo.ase...@gmail.com 
<mailto:natxo.ase...@gmail.com>> wrote:


hi,



On Mon, Sep 26, 2016 at 3:06 PM, Ludwig Krispenz
<lkris...@redhat.com <mailto:lkris...@redhat.com>> wrote:


On 09/26/2016 02:56 PM, Natxo Asenjo wrote:


so the command has not been successful in the kdc03. in the
dirsrv errors log  I see:

[26/Sep/2016:14:50:54 +0200] NSMMReplicationPlugin -
CleanAllRUV Task (rid 71): Not all replicas online, retrying
in 640 seconds...

this looks like there is still a replication agreement to one
of the no longer existing servers.

can you search for "... -b "cn=config"
"objectclass=nsds5replicationagreement"

and remove the ones no longer needed.


allow me to post the output of both commands as separate files

I am not really sure which one I need to remove.

--
Groeten,
natxo




--
--
Groeten,
natxo




--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] replicas removed, but incorrectly

[Ludwig Krispenz]

On 09/26/2016 02:56 PM, Natxo Asenjo wrote:



On Mon, Sep 26, 2016 at 1:54 PM, Natxo Asenjo <natxo.ase...@gmail.com 
<mailto:natxo.ase...@gmail.com>> wrote:





On Mon, Sep 26, 2016 at 1:50 PM, Ludwig Krispenz
<lkris...@redhat.com <mailto:lkris...@redhat.com>> wrote:


On 09/26/2016 01:36 PM, Natxo Asenjo wrote:

And in my example, the replica id would be 66, 96, 71 and 97, correct?

no, I don't think so. you searched 2 times the same host "-h
kdc04.unix.iriszorg.nl <http://kdc04.unix.iriszorg.nl>".
you need to search on kdc03 to find the current replicaid of
kdc03 and you have to keep it.



yes, you are right :(

 $ ldapsearch -Z -h kdc03.unix.iriszorg.nl
<http://kdc03.unix.iriszorg.nl> -D "cn=Directory Manager" -W -b
"o=ipaca"

"(&(objectclass=nstombstone)(nsUniqueId=---))"
| grep "nsds50ruv\|nsDS5ReplicaId"
Enter LDAP Password:
nsDS5ReplicaId: 66
nsds50ruv: {replicageneration} 50c1015c0060
nsds50ruv: {replica 66 ldap://kdc03.unix.iriszorg.nl:389
<http://kdc03.unix.iriszorg.nl:389>} 57e23f660042
nsds50ruv: {replica 1095 ldap://kdc04.unix.iriszorg.nl:389
<http://kdc04.unix.iriszorg.nl:389>} 57e4d75a044700
nsds50ruv: {replica 96 ldap://kdc01.unix.iriszorg.nl:7389
<http://kdc01.unix.iriszorg.nl:7389>} 50c1016c006
nsds50ruv: {replica 71 ldap://kdc03.unix.iriszorg.nl:389
<http://kdc03.unix.iriszorg.nl:389>} 57e140c70047
nsds50ruv: {replica 97 ldap://kdc02.unix.iriszorg.nl:7389
<http://kdc02.unix.iriszorg.nl:7389>} 50c101680061000


so I need to keep 66 and 1095, and run the task on 96, 71 and 97,
it would seem.

Thanks for spotting my error.



ok, so I have now run the commands against both ldap hosts (the kdc03 
and the kdc04), and now I have this:
you need to run it only against one host, it will propagate itself to 
the other replicas, if it can - see below.


 # ldapsearch -Z -h kdc04.unix.iriszorg.nl 
<http://kdc04.unix.iriszorg.nl> -D "cn=Directory Manager" -W -b 
"o=ipaca" 
"(&(objectclass=nstombstone)(nsUniqueId=---))" 
| grep "nsds50ruv\|nsDS5ReplicaId"

Enter LDAP Password:
nsDS5ReplicaId: 1095
nsds50ruv: {replicageneration} 50c1015c0060
nsds50ruv: {replica 1095 ldap://kdc04.unix.iriszorg.nl:389 
<http://kdc04.unix.iriszorg.nl:389>} 57e4d75a044700
nsds50ruv: {replica 66 ldap://kdc03.unix.iriszorg.nl:389 
<http://kdc03.unix.iriszorg.nl:389>} 57e23f660042


# ldapsearch -Z -h kdc03.unix.iriszorg.nl 
<http://kdc03.unix.iriszorg.nl> -D "cn=Directory Manager" -W -b 
"o=ipaca" 
"(&(objectclass=nstombstone)(nsUniqueId=---))" 
| grep "nsds50ruv\|nsDS5ReplicaId"

Enter LDAP Password:
nsDS5ReplicaId: 66
nsds50ruv: {replicageneration} 50c1015c0060
nsds50ruv: {replica 66 ldap://kdc03.unix.iriszorg.nl:389 
<http://kdc03.unix.iriszorg.nl:389>} 57e23f660042
nsds50ruv: {replica 1095 ldap://kdc04.unix.iriszorg.nl:389 
<http://kdc04.unix.iriszorg.nl:389>} 57e4d75a044700
nsds50ruv: {replica 96 ldap://kdc01.unix.iriszorg.nl:7389 
<http://kdc01.unix.iriszorg.nl:7389>} 50c1016c006
nsds50ruv: {replica 71 ldap://kdc03.unix.iriszorg.nl:389 
<http://kdc03.unix.iriszorg.nl:389>} 57e140c70047
nsds50ruv: {replica 97 ldap://kdc02.unix.iriszorg.nl:7389 
<http://kdc02.unix.iriszorg.nl:7389>} 50c101680061000


so the command has not been successful in the kdc03. in the dirsrv 
errors log  I see:


[26/Sep/2016:14:50:54 +0200] NSMMReplicationPlugin - CleanAllRUV Task 
(rid 71): Not all replicas online, retrying in 640 seconds...
this looks like there is still a replication agreement to one of the no 
longer existing servers.


can you search for "... -b "cn=config" 
"objectclass=nsds5replicationagreement"


and remove the ones no longer needed.
[26/Sep/2016:14:51:00 +0200] slapi_ldap_bind - Error: could not send 
startTLS request: error -1 (Can't contact LDAP server) errno 107 
(Transport endpoint is not connected)


but those replicas are gone (decommissioned). So how can I remove them?





--
regards,
Natxo





--
--
Groeten,
natxo




--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] replicas removed, but incorrectly

[Ludwig Krispenz]

On 09/26/2016 01:36 PM, Natxo Asenjo wrote:

hi,

I recently upgraded a centos 6.8 realm to centos 7.2 and it almost 
went correctly.


Now I see some errors in /var/log/dirsrv/slapd-INSTANCENAME/errors

26/Sep/2016:13:20:15 +0200] attrlist_replace - attr_replace 
(nsslapd-referral, ldap://kdc03.unix.iriszorg.nl:389/o%3Dipaca 
) failed


and according to 
http://www.freeipa.org/page/Troubleshooting#Replication_issues this 
points to a ruv problem.


So let's enumerate.

We had kdc01 replicating to kdc02 (both 6.8).

Then I created a replica from kdc01 to kdc03 (running 7.2).

And from kdc03 to kdc04 (both 7.2).

kdc01 and kdc02 are decommissioned, but kdc02 still shows in both 
kdc03 and kdc04:


$ ipa-replica-manage list
kdc02.unix.iriszorg.nl : master
kdc03.unix.iriszorg.nl : master
kdc04.unix.iriszorg.nl : master

and in

$ ipa-csreplica-manage list
Directory Manager password:
kdc02.unix.iriszorg.nl : master
kdc03.unix.iriszorg.nl : master
kdc04.unix.iriszorg.nl : master


>From kdc03:
$ ldapsearch -Z -h kdc04.unix.iriszorg.nl 
 -D "cn=Directory Manager" -W -b 
"o=ipaca" 
"(&(objectclass=nstombstone)(nsUniqueId=---))" 
| grep "nsds50ruv\|nsDS5ReplicaId"

Enter LDAP Password:
nsDS5ReplicaId: 1095
nsds50ruv: {replicageneration} 50c1015c0060
nsds50ruv: {replica 1095 ldap://kdc04.unix.iriszorg.nl:389 
} 57e4d75a044700
nsds50ruv: {replica 66 ldap://kdc03.unix.iriszorg.nl:389 
} 57e23f660042
nsds50ruv: {replica 96 ldap://kdc01.unix.iriszorg.nl:7389 
} 50c1016c006
nsds50ruv: {replica 71 ldap://kdc03.unix.iriszorg.nl:389 
} 57e140c70047
nsds50ruv: {replica 97 ldap://kdc02.unix.iriszorg.nl:7389 
} 50c101680061000


and from kdc04:

# ldapsearch -Z -h kdc04.unix.iriszorg.nl 
 -D "cn=Directory Manager" -W -b 
"o=ipaca" 
"(&(objectclass=nstombstone)(nsUniqueId=---))" 
| grep "nsds50ruv\|nsDS5ReplicaId"

Enter LDAP Password:
nsDS5ReplicaId: 1095
nsds50ruv: {replicageneration} 50c1015c0060
nsds50ruv: {replica 1095 ldap://kdc04.unix.iriszorg.nl:389 
} 57e4d75a044700
nsds50ruv: {replica 66 ldap://kdc03.unix.iriszorg.nl:389 
} 57e23f660042
nsds50ruv: {replica 96 ldap://kdc01.unix.iriszorg.nl:7389 
} 50c1016c006
nsds50ruv: {replica 71 ldap://kdc03.unix.iriszorg.nl:389 
} 57e140c70047
nsds50ruv: {replica 97 ldap://kdc02.unix.iriszorg.nl:7389 
} 50c101680061000



So now I have to run a clen ruv task like this (as seen in 
https://www.redhat.com/archives/freeipa-users/2016-May/msg00043.html):


# ldapmodify -ZZ -D "cn=directory manager" -W -a
dn: cn=clean 13, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: o=ipaca
replica-id: 13
cn: clean 13

And in my example, the replica id would be 66, 96, 71 and 97, correct?
no, I don't think so. you searched 2 times the same host "-h 
kdc04.unix.iriszorg.nl ".
you need to search on kdc03 to find the current replicaid of kdc03 and 
you have to keep it.

Thanks for confirming this, never done it before.
--
Groeten,
natxo




--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Server replication stopped working

[Ludwig Krispenz]

On 09/25/2016 09:35 PM, Youenn PIOLET wrote:

Hi there,

Same issue for me in a my 15 ipa-servers multi-master grid just after 
the update.

The replication is completely broken except on 3/15 nodes.

This is the second time I have to fully reinitialize the whole cluster 
for similar reason. I don't know what to do to clean this mess...
For more information: this cluster has been initialized on a fedora 
4.1.4 more than one year ago then complemetely migrated to Centos 7, 
IPA 4.2.

what is the exact version of 389-ds-base you are running ?

did these errors come out of the blue or are they related to some 
activities ? The csn which is not found has a timestamp of "Thu, 22 Sep 
2016 15:59:08 GMT" did anything happen around this time ?


Example on fr-master03 error logs:

[25/Sep/2016:19:27:31 +] NSMMReplicationPlugin - changelog program 
- agmt="cn=meTofr-master01.domain" (fr-master01:389): CSN 
57e3ffcc0003001a not found, we aren't as up to date, or we purged
[25/Sep/2016:19:27:31 +] NSMMReplicationPlugin - 
agmt="cn=meTofr-master01.domain" (fr-master01:389): Data required to 
update replica has been purged. The replica must be reinitialized.
[25/Sep/2016:19:27:31 +] NSMMReplicationPlugin - 
agmt="cn=meTofr-master01.domain" (fr-master01:389): Incremental update 
failed and requires administrator action

ipa: INFO: The ipactl command was successful
[25/Sep/2016:19:27:35 +] agmt="cn=meTofr-master02.domain" 
(fr-master02:389) - Can't locate CSN 57e3ffcc0003001a in the 
changelog (DB rc=-30988). If replication stops, the consumer may need 
to be reinitialized.
[25/Sep/2016:19:27:35 +] NSMMReplicationPlugin - changelog program 
- agmt="cn=meTofr-master02.domain" (fr-master02:389): CSN 
57e3ffcc0003001a not found, we aren't as up to date, or we purged
[25/Sep/2016:19:27:35 +] NSMMReplicationPlugin - 
agmt="cn=meTofr-master02.domain" (fr-master02:389): Data required to 
update replica has been purged. The replica must be reinitialized.
[25/Sep/2016:19:27:35 +] NSMMReplicationPlugin - 
agmt="cn=meTofr-master02.domain" (fr-master02:389): Incremental update 
failed and requires administrator action


Regards,

--
Youenn Piolet
piole...@gmail.com 
/
/

2016-09-23 17:51 GMT+02:00 Mike Driscoll >:


Hello.  I have four IPA servers replicating in full mesh.  All
four servers are running ipa-server-4.2.0-15.0.1.el7_2.19.x86_64.

This was working for some time but now I see that no replication
is occurring automatically at present.

When I update a user attribute on an IPA server, I see errors like
these:
[22/Sep/2016:16:53:49 -0700] attrlist_replace - attr_replace
(nsslapd-referral, ldap://ldap03.xx.com:389/o%3Dipaca) failed.
[22/Sep/2016:16:58:56 -0700] NSMMReplicationPlugin -
agmt="cn=masterAgreement1-ldap03.xx.com
-pki-tomcat" (ldap03:389):
Incremental update failed and requires administrator action

I can reinitialize without errors.
ipa-csreplica-manage re-initialize --from=ldap01.xx.com

ipa-replica-manage re-initialize --from=ldap01.xx.com

Afterwards I see my attribute (and other) changes are replicated
on each server I re-initialize from.  But subsequently,
replication doesn't seem to be happening.

I reinitialized according to the steps in Table 8.7, "Replication
Errors", but subsequent replication isn't occurring.  Any
suggestions?  Is it safe to identify one of my four servers as
containing up-to-date data, then sever and reinstate replication
relationships with the other three?

Mike






--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users

Go to http://freeipa.org for more info on the project






--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] FreeIPA upgrade from ipa-server-4.2.0-15.0.1.el7.centos.18 to ipa-server-4.2.0-15.0.1.el7.centos.19 (went sideways)

[Ludwig Krispenz]

On 09/23/2016 04:42 PM, Devin Acosta wrote:


Ludwig,

Thanks for that, for some reason I had to re-create the 
/var/lock/dirsrv/slapd-RSINC-LOCAL/server directory tree, it did not 
exist. Once I re-created it now the server starts. Should it have 
disappeared like that?
no. I don't know what was going during update and reboot. There have 
been cases when a dse.ldif was lost after crashing/rebooting a VM, but 
the missing lock directory is new to me.




On Fri, Sep 23, 2016 at 12:18 AM, Ludwig Krispenz <lkris...@redhat.com 
<mailto:lkris...@redhat.com>> wrote:


can you check if you have
/var/lock/dirsrv/slapd-RSINC-LOCAL

if the server user has permissions to write into this directory
and its subdirs or if any pid file still exists in
/var/lock/dirsrv/slapd-RSINC-LOCAL/server


On 09/23/2016 07:29 AM, Devin Acosta wrote:


Tonight,

I noticed there was like 30 packages to be applied on my IPA
server. I did the normal 'yum update' process and it completed. I
then rebooted the box for the new kernel to take affect and then
that is when IPA stopped working completely.

When I try to start the dirsrv@RSINC-LOCAL.service
<mailto:dirsrv@RSINC-LOCAL.service>, it throws up with:

[23/Sep/2016:05:19:38 +] - SSL alert: Configured NSS Ciphers
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_DHE_DSS_WITH_AES_256_GCM_SHA384: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_DHE_RSA_WITH_AES_256_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_DHE_DSS_WITH_AES_256_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_DHE_RSA_WITH_AES_256_CBC_SHA256: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_DHE_DSS_WITH_AES_256_CBC_SHA256: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_DHE_DSS_WITH_CAMELLIA_256_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_DHE_DSS_WITH_AES_128_GCM_SHA256: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_DHE_RSA_WITH_AES_128_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_DHE_DSS_WITH_AES_128_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_DHE_RSA_WITH_AES_128_CBC_SHA256: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_DHE_DSS_WITH_AES_128_CBC_SHA256: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_DHE_DSS_WITH_CAMELLIA_128_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_ECDH_RSA_WITH_AES_128_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_ECDH_RSA_WITH_AES_256_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_RSA_WITH_AES_256_GCM_SHA384: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_RSA_WITH_AES_256_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_RSA_WITH_AES_256_CBC_SHA256: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_RSA_WITH_CAMELLIA_256_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_RSA_WITH_AES_128_GCM_SHA256: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_RSA_WITH_AES_128_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert:
TLS_RSA_WITH_

Re: [Freeipa-users] FreeIPA upgrade from ipa-server-4.2.0-15.0.1.el7.centos.18 to ipa-server-4.2.0-15.0.1.el7.centos.19 (went sideways)

[Ludwig Krispenz]

can you check if you have
/var/lock/dirsrv/slapd-RSINC-LOCAL

if the server user has permissions to write into this directory and its 
subdirs or if any pid file still exists in 
/var/lock/dirsrv/slapd-RSINC-LOCAL/server


On 09/23/2016 07:29 AM, Devin Acosta wrote:


Tonight,

I noticed there was like 30 packages to be applied on my IPA server. I 
did the normal 'yum update' process and it completed. I then rebooted 
the box for the new kernel to take affect and then that is when IPA 
stopped working completely.


When I try to start the dirsrv@RSINC-LOCAL.service, it throws up with:

[23/Sep/2016:05:19:38 +] - SSL alert: Configured NSS Ciphers
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_DHE_DSS_WITH_AES_256_GCM_SHA384: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_DHE_RSA_WITH_AES_256_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_DHE_DSS_WITH_AES_256_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_DHE_RSA_WITH_AES_256_CBC_SHA256: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_DHE_DSS_WITH_AES_256_CBC_SHA256: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_DHE_DSS_WITH_CAMELLIA_256_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_DHE_DSS_WITH_AES_128_GCM_SHA256: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_DHE_RSA_WITH_AES_128_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_DHE_DSS_WITH_AES_128_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_DHE_RSA_WITH_AES_128_CBC_SHA256: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_DHE_DSS_WITH_AES_128_CBC_SHA256: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_DHE_DSS_WITH_CAMELLIA_128_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_ECDH_RSA_WITH_AES_128_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_ECDH_RSA_WITH_AES_256_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_RSA_WITH_AES_256_GCM_SHA384: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_RSA_WITH_AES_256_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_RSA_WITH_AES_256_CBC_SHA256: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_RSA_WITH_CAMELLIA_256_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_RSA_WITH_AES_128_GCM_SHA256: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_RSA_WITH_AES_128_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_RSA_WITH_AES_128_CBC_SHA256: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: 
TLS_RSA_WITH_CAMELLIA_128_CBC_SHA: enabled
[23/Sep/2016:05:19:38 +] - SSL alert: TLS_RSA_WITH_SEED_CBC_SHA: 
enabled
[23/Sep/2016:05:19:38 +] SSL Initialization - Configured SSL 
version range: min: TLS1.0, max: TLS1.2
[23/Sep/2016:05:19:38 +] - Shutting down due to possible conflicts 
with other slapd processes


*I am not sure what to do about the error "Shutting down due to 
possible conflicts with other slapd processes"??*
The dirserv won't start, and therefore IPA won't start either. Is 
there some way to do some cleanup or to have it repair the issue?


Any help is greatly appreciated!!!

Devin.







--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

-- 
Manage your subscription for the 

Re: [Freeipa-users] Freeipa 4.2.0 hangs intermittently

[Ludwig Krispenz]

Hi,
On 09/13/2016 07:37 PM, Rakesh Rajasekharan wrote:

Hi All,

Have finally made some progress with this.. after changing the 
checkpoint interval to 180, my hangs have gone down now..


However, I faced a similar hang yesterday... users were not able to 
login.. , though this time the ns-slapd did not had any issues and 
ldapsearch worked fine possibly due to the changes in checpoint. So, I 
think I hit some other issue this time


this is a bit confusing, if your server crashes with the attached 
stacktrace ldapsearch cannot work.


About the core, it looks like you are hitting this  issue: 
https://fedorahosted.org/389/ticket/48388


I had a core genrated and this is the stacktrace of it.. can you 
please go through this and help me identify what could be causing the 
issue this time.. I have put in lot of efforts to debug and really 
would love to have this working in my prod env.. as it does in my 
other envs...


GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 


This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /usr/sbin/ns-slapd...
warning: the debug information found in 
"/usr/lib/debug//usr/sbin/ns-slapd.debug" does not match 
"/usr/sbin/ns-slapd" (CRC mismatch).



warning: the debug information found in 
"/usr/lib/debug/usr/sbin/ns-slapd.debug" does not match 
"/usr/sbin/ns-slapd" (CRC mismatch).


Reading symbols from /usr/sbin/ns-slapd...(no debugging symbols 
found)...done.

(no debugging symbols found)...done.
[New LWP 15255]
[New LWP 15286]
[New LWP 15245]
[New LWP 15246]
[New LWP 15247]
[New LWP 15248]
[New LWP 15243]

warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/libslapd.so.0.0.0.debug" does not 
match "/usr/lib64/dirsrv/libslapd.so.0" (CRC mismatch).



warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/libslapd.so.0.0.0.debug" does not 
match "/usr/lib64/dirsrv/libslapd.so.0" (CRC mismatch).


[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libsyntax-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libsyntax-plugin.so" (CRC 
mismatch).



warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libsyntax-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libsyntax-plugin.so" (CRC 
mismatch).



warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libbitwise-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libbitwise-plugin.so" (CRC 
mismatch).



warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libbitwise-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libbitwise-plugin.so" (CRC 
mismatch).


...skipping...
-rw---. 1 dirsrv dirsrv  0 Sep  8 02:55 audit
-rw---. 1 dirsrv dirsrv 2551824384 Sep 12 17:32 core.10450
-rw---. 1 dirsrv dirsrv 1464463360 Sep 12 19:35 core.14709
-rw---. 1 dirsrv dirsrv 4483862528 Sep 13 01:05 core.15243
-rw---. 1 dirsrv dirsrv   66288165 Sep 13 02:10 errors
-rw---. 1 dirsrv dirsrv  104964391 Sep 13 08:30 access.20160913-074214
-rw---. 1 dirsrv dirsrv  105021859 Sep 13 09:26 access.20160913-083046
-rw---. 1 dirsrv dirsrv  104861746 Sep 13 10:31 access.20160913-092646
-rw---. 1 dirsrv dirsrv  105069140 Sep 13 11:36 access.20160913-103137
-rw---. 1 dirsrv dirsrv  104913480 Sep 13 12:41 access.20160913-113638
-rw---. 1 dirsrv dirsrv  105186788 Sep 13 13:46 access.20160913-124118
-rw---. 1 dirsrv dirsrv  105162159 Sep 13 14:51 access.20160913-134619
-rw---. 1 dirsrv dirsrv  105256624 Sep 13 15:56 access.20160913-145120
-rw---. 1 dirsrv dirsrv  105231158 Sep 13 17:01 access.20160913-155620
-rw---. 1 dirsrv dirsrv   1044 Sep 13 17:01 access.rotationinfo
-rw-r--r--. 1 root   root19287 Sep 13 17:28 
stacktrace.1473787719.txt

-rw---. 1 dirsrv dirsrv   45608914 Sep 13 17:29 access
[root@prod-ipa-master-int slapd-SPRINKLR-COM]# gdb -ex 'set confirm 
off' -ex 'set pagination off' -ex 'thread apply all bt full' -ex 
'quit' /usr/sbin/ns-slapd 
/var/log/dirsrv/slapd-SPRINKLR-COM/core.15243 stacktrace.`date 
+%s`.txt 2>&1^C
[root@prod-ipa-master-int slapd-SPRINKLR-COM]# gdb -ex 'set confirm 
off' -ex 'set pagination off' -ex 'thread apply all bt full' -ex 
'quit' /usr/sbin/ns-slapd 
/var/log/dirsrv/slapd-SPRINKLR-COM/core.15243 > stacktrace.`date 
+%s`.txt 2>&1

[root@prod-ipa-master-int slapd-SPRINKLR-COM]# ls -ltr
total 6404952
-rw---. 1 dirsrv dirsrv   

Re: [Freeipa-users] Two masters and one of them is desynchronized

[Ludwig Krispenz]

On 08/25/2016 04:41 PM, bahan w wrote:


Hello everyone.

Could you explain to me about this field Sent/Skipped please ?
if replication is enabled all changes on a server are logged into the 
changelog -changes coming from clients and internal changes (eg mmeberof 
update, passwordpolocy updates, lstlogin time ...).
In the replication agreement you can configure attributes for which 
changes are not replicated (keep them local) - and IPA uses this feature 
eg for krblastlogintime.


Looking at the replication traffic your monitoring shows, I think most 
of the "real" updates are going to one server and most of the clients 
triggering internal updates are going to the other. This makes 
replciation in one direction "normal" and in the other fractional. The 
problem with fractional is that the determined staring point for a 
replciation session can b every far behind and it again and again 
iterates over the same changes until finally an update which is not 
skipped is found.


There are some options to improve this:
- upgarde to a newer version, teh DS will automatically generate updates 
to a "keep alive" entry, so that the sequences of skipped changes get 
much smaller
- do it yourself by regularily applying a dummy update on the 
problematic server which will be replicated
- check configuration if writing the internal mods can be avoided, I 
think there is an option not to log krblastlogin




I checked the doc and found this :
###

Sent/Skipped :



The number of changes that were sent from the supplier and the number 
skipped in the replication update. The numbers are kept in suppliers’ 
memory only and are cleared if the supplier is restarted.


###

If I check the first part :
###
Master: :389 ldap://:389/
Replica ID: 4
Replica Root: dc=
Max CSN: 57bdcd3600010004 (08/24/2016 18:37:10 1 0)
Receiver: :389 ldap://:389/
Type: master
Time Lag: 0:00:00
Max CSN: 57bdcd3600010004 (08/24/2016 18:37:10 1 0)
Last Modify Time: 8/24/2016 18:36:32
Supplier: :389
Sent/Skipped: 182110 / 1054
Update Status: 0 Replica acquired successfully: Incremental update 
succeeded

Update Started: 08/24/2016 18:36:32
Update Ended: 08/24/2016 18:36:34
Schedule: always in sync
SSL: SASL/GSSAPI
###

This is the replication from the MASTER OK (the supplier) to the 
MASTER UNSYNC (the receiver), right ?

So, the MASTER OK sent 182110 changes.
And in addition to these 182110 changes, 1054 changes were sent to the 
MASTER UNSYNC but skipped by the MASTER UNSYNC, right ?

Why are they skipped ?

In the other side, if I take the second part :
###
Master: :389 ldap://:389/
Replica ID: 3
Replica Root: dc=
Max CSN: 57bdbda10003 (08/24/2016 17:30:41)
Receiver: :389 ldap://:389/
Type: master
Time Lag: - 0:22:29
Max CSN: 57bdb85c0003 (08/24/2016 17:08:12)
Last Modify Time: 8/24/2016 17:07:34
Supplier: :389
Sent/Skipped: 3 / 9048655
Update Status: 0 Replica acquired successfully: Incremental update 
succeeded

Update Started: 08/24/2016 18:36:33
Update Ended: 08/24/2016 18:36:34
Schedule: always in sync
SSL: SASL/GSSAPI
###

The supplier is the MASTER UNSYNC and the receiver is the MASTER OK.
In this case I have only 3 changes sent.
And in addition to these 3 changes, 9 048 655 changes were sent but 
skipped on the MASTER OK, right ?


I ask these questions just to be sure I understand right the return of 
the pl script.



Best regards.

Bahan


--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Two masters and one of them is desynchronized

[Ludwig Krispenz]

I just noticed that you have many skipped entries, Sent/Skipped: 3 / 9045345

that could be an effect of fractional replication which  reiterates the 
same sequence of changes. This is fixed in recent releases, but looks 
like your on RHEL 6.6


Ludwig

On 08/24/2016 06:33 PM, bahan w wrote:

Hey guys.

I performed it :
###
# /usr/bin/repl-monitor.pl  -f /tmp/checkconf -s
Directory Server Replication Status (Version 1.1)

Time: Wed Aug 24 2016 18:16:50

Master: :389 ldap://:389/
Replica ID: 4
Replica Root: dc=
Max CSN: 57bdc89700030004 (08/24/2016 18:17:27 3 0)
Receiver: :389 ldap://:389/
Type: master
Time Lag: 0:00:00
Max CSN: 57bdc89700030004 (08/24/2016 18:17:27 3 0)
Last Modify Time: 8/24/2016 18:16:50
Supplier: :389
Sent/Skipped: 179031 / 1037
Update Status: 0 Replica acquired successfully: Incremental update started
Update Started: 08/24/2016 18:16:50
Update Ended: n/a
Schedule: always in sync
SSL: SASL/GSSAPI

Master: :389 ldap://:389/
Replica ID: 3
Replica Root: dc=
Max CSN: 57bdbda10003 (08/24/2016 17:30:41)
Receiver: :389 ldap://:389/
Type: master
Time Lag: - 0:22:29
Max CSN: 57bdb85c0003 (08/24/2016 17:08:12)
Last Modify Time: 8/24/2016 17:07:34
Supplier: :389
Sent/Skipped: 3 / 9045345
Update Status: 0 Replica acquired successfully: Incremental update started
Update Started: 08/24/2016 18:16:50
Update Ended: n/a
Schedule: always in sync
SSL: SASL/GSSAPI
###

Do you see something strange in there ?
I have another environment where I have two replicated master and they 
are OK.

And when I check the same command, the result is a little bit different :
###
Master: :389 ldap://:389/
Replica ID: 4
Replica Root: dc=
Max CSN: 57bdc88d00030004 (08/24/2016 18:17:17 3 0)
Receiver: :389 ldap://:389/
Type: master
Time Lag: 0:00:00
Max CSN: 57bdc88d00030004 (08/24/2016 18:17:17 3 0)
Last Modify Time: 8/24/2016 18:16:00
Supplier: :389
Sent/Skipped: 343515 / 0
Update Status: 0 Replica acquired successfully: Incremental update 
succeeded

Update Started: 08/24/2016 18:15:59
Update Ended: 08/24/2016 18:16:08
Schedule: always in sync
SSL: SASL/GSSAPI

Master: :389 ldap://:389/
Replica ID: 3
Replica Root: dc=
Max CSN: 57bdc88700080003 (08/24/2016 18:17:11 8 0)
Receiver: :389 ldap://:389/
Type: master
Time Lag: - 390:51:38
Max CSN: 57a8500d00040003 (08/08/2016 11:25:33 4 0)
Last Modify Time: 8/8/2016 11:24:28
Supplier: :389
Sent/Skipped: 5 / 2596073
Update Status: 0 Replica acquired successfully: Incremental update 
succeeded

Update Started: 08/24/2016 18:16:00
Update Ended: 08/24/2016 18:16:12
Schedule: always in sync
SSL: SASL/GSSAPI
###

Best regards.

Bahan


--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Two masters and one of them is desynchronized

[Ludwig Krispenz]

The replication agreements to the "unsync" master says that update has 
started, so it looks like replication connection is active.
You need to check the access and error logs of bot sides and check if 
tehre is replication traffic


On 08/24/2016 06:33 PM, bahan w wrote:

Hey guys.

I performed it :
###
# /usr/bin/repl-monitor.pl  -f /tmp/checkconf -s
Directory Server Replication Status (Version 1.1)

Time: Wed Aug 24 2016 18:16:50

Master: :389 ldap://:389/
Replica ID: 4
Replica Root: dc=
Max CSN: 57bdc89700030004 (08/24/2016 18:17:27 3 0)
Receiver: :389 ldap://:389/
Type: master
Time Lag: 0:00:00
Max CSN: 57bdc89700030004 (08/24/2016 18:17:27 3 0)
Last Modify Time: 8/24/2016 18:16:50
Supplier: :389
Sent/Skipped: 179031 / 1037
Update Status: 0 Replica acquired successfully: Incremental update started
Update Started: 08/24/2016 18:16:50
Update Ended: n/a
Schedule: always in sync
SSL: SASL/GSSAPI

Master: :389 ldap://:389/
Replica ID: 3
Replica Root: dc=
Max CSN: 57bdbda10003 (08/24/2016 17:30:41)
Receiver: :389 ldap://:389/
Type: master
Time Lag: - 0:22:29
Max CSN: 57bdb85c0003 (08/24/2016 17:08:12)
Last Modify Time: 8/24/2016 17:07:34
Supplier: :389
Sent/Skipped: 3 / 9045345
Update Status: 0 Replica acquired successfully: Incremental update started
Update Started: 08/24/2016 18:16:50
Update Ended: n/a
Schedule: always in sync
SSL: SASL/GSSAPI
###

Do you see something strange in there ?
I have another environment where I have two replicated master and they 
are OK.

And when I check the same command, the result is a little bit different :
###
Master: :389 ldap://:389/
Replica ID: 4
Replica Root: dc=
Max CSN: 57bdc88d00030004 (08/24/2016 18:17:17 3 0)
Receiver: :389 ldap://:389/
Type: master
Time Lag: 0:00:00
Max CSN: 57bdc88d00030004 (08/24/2016 18:17:17 3 0)
Last Modify Time: 8/24/2016 18:16:00
Supplier: :389
Sent/Skipped: 343515 / 0
Update Status: 0 Replica acquired successfully: Incremental update 
succeeded

Update Started: 08/24/2016 18:15:59
Update Ended: 08/24/2016 18:16:08
Schedule: always in sync
SSL: SASL/GSSAPI

Master: :389 ldap://:389/
Replica ID: 3
Replica Root: dc=
Max CSN: 57bdc88700080003 (08/24/2016 18:17:11 8 0)
Receiver: :389 ldap://:389/
Type: master
Time Lag: - 390:51:38
Max CSN: 57a8500d00040003 (08/08/2016 11:25:33 4 0)
Last Modify Time: 8/8/2016 11:24:28
Supplier: :389
Sent/Skipped: 5 / 2596073
Update Status: 0 Replica acquired successfully: Incremental update 
succeeded

Update Started: 08/24/2016 18:16:00
Update Ended: 08/24/2016 18:16:12
Schedule: always in sync
SSL: SASL/GSSAPI
###

Best regards.

Bahan


--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] clean-ruv

[Ludwig Krispenz]

On 08/24/2016 01:08 AM, Ian Harding wrote:


On 08/23/2016 03:14 AM, Ludwig Krispenz wrote:

On 08/23/2016 11:52 AM, Ian Harding wrote:

Ah.  I see.  I mixed those up but I see that those would have to be
consistent.

However, I have been trying to beat some invalid RUV to death for a long
time and I can't seem to kill them.

For example, bellevuenfs has 9 and 16 which are invalid:

[ianh@seattlenfs ~]$ ldapsearch -ZZ -h seattlenfs.bpt.rocks -D
"cn=Directory Manager" -W -b "dc=bpt,dc=rocks"
"(&(objectclass=nstombstone)(nsUniqueId=---))"

| grep "nsds50ruv\|nsDS5ReplicaId"
Enter LDAP Password:
nsDS5ReplicaId: 7
nsds50ruv: {replicageneration} 55c8f3640004
nsds50ruv: {replica 7 ldap://seattlenfs.bpt.rocks:389}
568ac3cc0007 57
nsds50ruv: {replica 20 ldap://freeipa-sea.bpt.rocks:389}
57b1037700020014
nsds50ruv: {replica 18 ldap://bpt-nyc1-nfs.bpt.rocks:389}
57a4780100010012
nsds50ruv: {replica 15 ldap://fremontnis.bpt.rocks:389}
57a40386000f 5
nsds50ruv: {replica 14 ldap://freeipa-dal.bpt.rocks:389}
57a2dccd000e
nsds50ruv: {replica 17 ldap://edinburghnfs.bpt.rocks:389}
57a422f90011
nsds50ruv: {replica 19 ldap://bellevuenfs.bpt.rocks:389}
57a4f20d00060013
nsds50ruv: {replica 16 ldap://bellevuenfs.bpt.rocks:389}
57a417060010
nsds50ruv: {replica 9 ldap://bellevuenfs.bpt.rocks:389}
570484ee0009 5


So I try to kill them like so:
[ianh@seattlenfs ~]$ ipa-replica-manage clean-ruv 9 --force --cleanup
ipa: WARNING: session memcached servers not running
Clean the Replication Update Vector for bellevuenfs.bpt.rocks:389

Cleaning the wrong replica ID will cause that server to no
longer replicate so it may miss updates while the process
is running. It would need to be re-initialized to maintain
consistency. Be very careful.
Background task created to clean replication data. This may take a while.
This may be safely interrupted with Ctrl+C
^C[ianh@seattlenfs ~]$ ipa-replica-manage clean-ruv 16 --force --cleanup
ipa: WARNING: session memcached servers not running
Clean the Replication Update Vector for bellevuenfs.bpt.rocks:389

Cleaning the wrong replica ID will cause that server to no
longer replicate so it may miss updates while the process
is running. It would need to be re-initialized to maintain
consistency. Be very careful.
Background task created to clean replication data. This may take a while.
This may be safely interrupted with Ctrl+C
^C[ianh@seattlenfs ~]$ ipa-replica-manage list-clean-ruv
ipa: WARNING: session memcached servers not running
CLEANALLRUV tasks
RID 16: Waiting to process all the updates from the deleted replica...
RID 9: Waiting to process all the updates from the deleted replica...

No abort CLEANALLRUV tasks running
[ianh@seattlenfs ~]$ ipa-replica-manage list-clean-ruv
ipa: WARNING: session memcached servers not running
CLEANALLRUV tasks
RID 16: Waiting to process all the updates from the deleted replica...
RID 9: Waiting to process all the updates from the deleted replica...

and it never finishes.

seattlenfs is the first master, that's the only place I should have to
run this command, right?

right, you need to run it only on one master, but this ease of use can
become the problem.
The cleanallruv task is propagated to all servers in the topology and it
does this based on the replication agreements it finds.
A frequent cause of failure is that replication agreements still exist
pointing to no longer existing servers. It is a bit tedious, but could
you run the following search on ALL
of your current replicas (as directory manager):

ldapsearch .. -b "cn=config" "objectclass=nsds5replicationagreement"
nsds5replicahost

if you find any agreement where nsds5replicahost is a host no longer
existing or working, delete these agreements.

I have 7 FreeIPA servers, all of which have been in existence in some
form or another since I started.  It used to work great.  I've broken it
now but the hostnames and ip addresses all still exist.  I've
uninstalled and reinstalled them a few times which I think is the source
of my troubles so I tried to straighten out the RUVs and probably messed
that up pretty good

Anyway, now what I THINK I have is

seattlenfs
|-freeipa-sea
   |- freeipa-dal
   |- bellevuenfs
   |- fremontnis
   |- bpt-nyc1-nfs
   |- edinburghnfs

Until I get this squared away I've turned off ipa services on all but
seattlenfs, freeipa-sea and freeipa-dal and am hoping that any password
changes etc. happen on seattlenfs.  I need the other two because they
are my DNS.  The rest I can kind of live without since they are just
local instances living on nfs servers.

Here's the output from that ldap query on all the hosts:

yes, looks like the replication agreements are fine, but the RUVs are not.

In the o=ipaca suffix, there is a reference to bellvuenis:

 [{replica 76
ldap://bellevuenis.bpt.rocks:389} 56f385eb0007004c


but t

Re: [Freeipa-users] clean-ruv

[Ludwig Krispenz]

On 08/23/2016 11:52 AM, Ian Harding wrote:

Ah.  I see.  I mixed those up but I see that those would have to be
consistent.

However, I have been trying to beat some invalid RUV to death for a long
time and I can't seem to kill them.

For example, bellevuenfs has 9 and 16 which are invalid:

[ianh@seattlenfs ~]$ ldapsearch -ZZ -h seattlenfs.bpt.rocks -D
"cn=Directory Manager" -W -b "dc=bpt,dc=rocks"
"(&(objectclass=nstombstone)(nsUniqueId=---))"
| grep "nsds50ruv\|nsDS5ReplicaId"
Enter LDAP Password:
nsDS5ReplicaId: 7
nsds50ruv: {replicageneration} 55c8f3640004
nsds50ruv: {replica 7 ldap://seattlenfs.bpt.rocks:389}
568ac3cc0007 57
nsds50ruv: {replica 20 ldap://freeipa-sea.bpt.rocks:389}
57b1037700020014
nsds50ruv: {replica 18 ldap://bpt-nyc1-nfs.bpt.rocks:389}
57a4780100010012
nsds50ruv: {replica 15 ldap://fremontnis.bpt.rocks:389}
57a40386000f 5
nsds50ruv: {replica 14 ldap://freeipa-dal.bpt.rocks:389}
57a2dccd000e
nsds50ruv: {replica 17 ldap://edinburghnfs.bpt.rocks:389}
57a422f90011
nsds50ruv: {replica 19 ldap://bellevuenfs.bpt.rocks:389}
57a4f20d00060013
nsds50ruv: {replica 16 ldap://bellevuenfs.bpt.rocks:389}
57a417060010
nsds50ruv: {replica 9 ldap://bellevuenfs.bpt.rocks:389}
570484ee0009 5


So I try to kill them like so:
[ianh@seattlenfs ~]$ ipa-replica-manage clean-ruv 9 --force --cleanup
ipa: WARNING: session memcached servers not running
Clean the Replication Update Vector for bellevuenfs.bpt.rocks:389

Cleaning the wrong replica ID will cause that server to no
longer replicate so it may miss updates while the process
is running. It would need to be re-initialized to maintain
consistency. Be very careful.
Background task created to clean replication data. This may take a while.
This may be safely interrupted with Ctrl+C
^C[ianh@seattlenfs ~]$ ipa-replica-manage clean-ruv 16 --force --cleanup
ipa: WARNING: session memcached servers not running
Clean the Replication Update Vector for bellevuenfs.bpt.rocks:389

Cleaning the wrong replica ID will cause that server to no
longer replicate so it may miss updates while the process
is running. It would need to be re-initialized to maintain
consistency. Be very careful.
Background task created to clean replication data. This may take a while.
This may be safely interrupted with Ctrl+C
^C[ianh@seattlenfs ~]$ ipa-replica-manage list-clean-ruv
ipa: WARNING: session memcached servers not running
CLEANALLRUV tasks
RID 16: Waiting to process all the updates from the deleted replica...
RID 9: Waiting to process all the updates from the deleted replica...

No abort CLEANALLRUV tasks running
[ianh@seattlenfs ~]$ ipa-replica-manage list-clean-ruv
ipa: WARNING: session memcached servers not running
CLEANALLRUV tasks
RID 16: Waiting to process all the updates from the deleted replica...
RID 9: Waiting to process all the updates from the deleted replica...

and it never finishes.

seattlenfs is the first master, that's the only place I should have to
run this command, right?
right, you need to run it only on one master, but this ease of use can 
become the problem.
The cleanallruv task is propagated to all servers in the topology and it 
does this based on the replication agreements it finds.
A frequent cause of failure is that replication agreements still exist 
pointing to no longer existing servers. It is a bit tedious, but could 
you run the following search on ALL

of your current replicas (as directory manager):

ldapsearch .. -b "cn=config" "objectclass=nsds5replicationagreement" 
nsds5replicahost


if you find any agreement where nsds5replicahost is a host no longer 
existing or working, delete these agreements.


I'm about to burn everything down and ipa-server-install --uninstall but
I've done that before a couple times and that seems to be what got me
into this mess...

Thank you for your help.




On 08/23/2016 01:37 AM, Ludwig Krispenz wrote:

looks like you are searching the nstombstone below "o=ipaca", but you
are cleaning ruvs in "dc=bpt,dc=rocks",

your attrlist_replace error refers to the bpt,rocks backend, so you
should search the tombstone entry ther, then determine which replicaIDs
to remove.

Ludwig

On 08/23/2016 09:20 AM, Ian Harding wrote:

I've followed the procedure in this thread:

https://www.redhat.com/archives/freeipa-users/2016-May/msg00043.html

and found my list of RUV that don't have an existing replica id.

I've tried to remove them like so:

[root@seattlenfs ianh]# ldapmodify -D "cn=directory manager" -W -a
Enter LDAP Password:
dn: cn=clean 97, cn=cleanallruv, cn=tasks, cn=config
objectclass: top
objectclass: extensibleObject
replica-base-dn: dc=bpt,dc=rocks
replica-id: 97
replica-force-cleaning: yes
cn: clean 97

adding new entry "cn=clean 97, cn=cleanallruv, cn=tasks, cn=config"

[root@seattlenfs ianh]# ipa-replica-ma

Re: [Freeipa-users] clean-ruv

[Ludwig Krispenz]

looks like you are searching the nstombstone below "o=ipaca", but you 
are cleaning ruvs in "dc=bpt,dc=rocks",


your attrlist_replace error refers to the bpt,rocks backend, so you 
should search the tombstone entry ther, then determine which replicaIDs 
to remove.


Ludwig

On 08/23/2016 09:20 AM, Ian Harding wrote:

I've followed the procedure in this thread:

https://www.redhat.com/archives/freeipa-users/2016-May/msg00043.html

and found my list of RUV that don't have an existing replica id.

I've tried to remove them like so:

[root@seattlenfs ianh]# ldapmodify -D "cn=directory manager" -W -a
Enter LDAP Password:
dn: cn=clean 97, cn=cleanallruv, cn=tasks, cn=config
objectclass: top
objectclass: extensibleObject
replica-base-dn: dc=bpt,dc=rocks
replica-id: 97
replica-force-cleaning: yes
cn: clean 97

adding new entry "cn=clean 97, cn=cleanallruv, cn=tasks, cn=config"

[root@seattlenfs ianh]# ipa-replica-manage list-clean-ruv
CLEANALLRUV tasks
RID 9: Waiting to process all the updates from the deleted replica...
RID 96: Successfully cleaned rid(96).
RID 97: Successfully cleaned rid(97).

No abort CLEANALLRUV tasks running


and yet, they are still there...

[root@seattlenfs ianh]# ldapsearch -ZZ -h seattlenfs.bpt.rocks -D
"cn=Directory Manager" -W -b "o=ipaca"
"(&(objectclass=nstombstone)(nsUniqueId=---))"
| grep "nsds50ruv\|nsDS5ReplicaId"
Enter LDAP Password:
nsDS5ReplicaId: 81
nsds50ruv: {replicageneration} 55c8f3ae0060
nsds50ruv: {replica 81 ldap://seattlenfs.bpt.rocks:389}
568ac4310051 5
nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389}
57b103d40429000
nsds50ruv: {replica 1070 ldap://bellevuenfs.bpt.rocks:389}
57a4f270042e000
nsds50ruv: {replica 1075 ldap://bpt-nyc1-nfs.bpt.rocks:389}
57a47865043300
nsds50ruv: {replica 1080 ldap://bellevuenfs.bpt.rocks:389}
57a417670438000
nsds50ruv: {replica 1085 ldap://fremontnis.bpt.rocks:389}
57a403e6043d
nsds50ruv: {replica 1090 ldap://freeipa-dal.bpt.rocks:389}
57a2dd350442000
nsds50ruv: {replica 1095 ldap://freeipa-sea.bpt.rocks:389}
579a963c0447000
nsds50ruv: {replica 96 ldap://freeipa-sea.bpt.rocks:389}
55c8f3bd0060
nsds50ruv: {replica 86 ldap://fremontnis.bpt.rocks:389}
5685b24e0056 5
nsds50ruv: {replica 91 ldap://seattlenis.bpt.rocks:389}
567ad6180001005b 5
nsds50ruv: {replica 97 ldap://freeipa-dal.bpt.rocks:389}
55c8f3ce0061
nsds50ruv: {replica 76 ldap://bellevuenis.bpt.rocks:389}
56f385eb0007004c
nsds50ruv: {replica 71 ldap://bellevuenfs.bpt.rocks:389}
570485690047
nsds50ruv: {replica 66 ldap://bpt-nyc1-nfs.bpt.rocks:389}
5733e594000a0042
nsds50ruv: {replica 61 ldap://edinburghnfs.bpt.rocks:389}
57442125003d
nsds50ruv: {replica 1195 ldap://edinburghnfs.bpt.rocks:389}
57a4239004ab00

What have I done wrong?

The problem I am trying to solve is that seattlenfs.bpt.rocks sends
updates to all its children, but their changes don't come back because
of these errors:

[23/Aug/2016:00:02:16 -0700] attrlist_replace - attr_replace
(nsslapd-referral,
ldap://seattlenfs.bpt.rocks:389/dc%3Dbpt%2Cdc%3Drocks) failed.

in effect, the replication agreements are one-way.

Any ideas?

- Ian



--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] replica_generate_next_csn messages in dirsrv error logs

[Ludwig Krispenz]

Thanks,

I looked into the logs, I think the messages are harmless, just an 
effect of csn adjustment due to time difference on the two machines. I 
had said that the replication protocol will try to adjust the csn 
generator, but looks like you have long lasting replication connections 
and the adjustment is done only at the beginning. Maybe we can look into 
this and improve it.

Just the tracking of one of these error messages:

the entry is modified on adm3
adm3 :[19/Aug/2016:15:47:05 -0400] conn=13 op=126637 MOD 
dn="idnsname=rc.usf.edu,cn=dns,dc=rc,dc=usf,dc=edu"
adm3 :[19/Aug/2016:15:47:05 -0400] conn=13 op=126637 RESULT err=0 
tag=103 nentries=0 etime=0 csn=57b763030016

this mod is replicated to adm0
adm0 :[19/Aug/2016:15:47:06 -0400] conn=1395 op=86121 MOD 
dn="idnsname=rc.usf.edu,cn=dns,dc=rc,dc=usf,dc=edu"
adm0 :[19/Aug/2016:15:47:06 -0400] conn=1395 op=86121 RESULT err=0 
tag=103 nentries=0 etime=0 csn=57b763030016

the entry is modified again on adm0
adm0 :[19/Aug/2016:15:47:07 -0400] conn=27 op=1108697 MOD 
dn="idnsname=rc.usf.edu,cn=dns,dc=rc,dc=usf,dc=edu"
but it looks like the csn generated is smaller than the one already in 
the entry, and it is adjusted
adm0 :[19/Aug/2016:15:47:07 -0400] - replica_generate_next_csn: 
opcsn=57b76301000a0004 <= basecsn=57b763030016, adjusted 
opcsn=57b7630300010004

then the result is logged with the adjusted csn
adm0 :[19/Aug/2016:15:47:07 -0400] conn=27 op=1108697 RESULT err=0 
tag=103 nentries=0 etime=0 csn=57b7630300010004


so the mechanism works, but the messages may be confusing and 
improvement of the protocol could be investigated.


One question I have, but someone more familiar with dns should answer:
we  have regular updates of the same entry on both replicas, about every 
2 seconds, what is the reason for this ?



/tmp/adm3-logs-del39-errors.txt:[19/Aug/2016:15:47:03 -0400] conn=13 
op=126630 MOD dn="idnsname=rc.usf.edu,cn=dns,dc=rc,dc=usf,dc=edu"
/tmp/adm3-logs-del39-errors.txt:[19/Aug/2016:15:47:05 -0400] conn=13 
op=126637 MOD dn="idnsname=rc.usf.edu,cn=dns,dc=rc,dc=usf,dc=edu"
/tmp/adm3-logs-del39-errors.txt:[19/Aug/2016:15:47:07 -0400] conn=13 
op=126646 MOD dn="idnsname=rc.usf.edu,cn=dns,dc=rc,dc=usf,dc=edu"
/tmp/adm3-logs-del39-errors.txt:[19/Aug/2016:15:47:09 -0400] conn=13 
op=126653 MOD dn="idnsname=rc.usf.edu,cn=dns,dc=rc,dc=usf,dc=edu"
/tmp/adm3-logs-del39-errors.txt:[19/Aug/2016:15:47:13 -0400] conn=13 
op=12 MOD dn="idnsname=rc.usf.edu,cn=dns,dc=rc,dc=usf,dc=edu"
/tmp/adm3-logs-del39-errors.txt:[19/Aug/2016:15:47:16 -0400] conn=13 
op=126673 MOD dn="idnsname=rc.usf.edu,cn=dns,dc=rc,dc=usf,dc=edu"
/tmp/adm3-logs-del39-errors.txt:[19/Aug/2016:15:47:18 -0400] conn=13 
op=126689 MOD dn="idnsname=rc.usf.edu,cn=dns,dc=rc,dc=usf,dc=edu"
/tmp/adm3-logs-del39-errors.txt:[19/Aug/2016:15:47:20 -0400] conn=13 
op=126696 MOD dn="idnsname=rc.usf.edu,cn=dns,dc=rc,dc=usf,dc=edu"
/tmp/adm3-logs-del39-errors.txt:[19/Aug/2016:15:47:21 -0400] conn=13 
op=126702 MOD dn="idnsname=rc.usf.edu,cn=dns,dc=rc,dc=usf,dc=edu"
/tmp/adm3-logs-del39-errors.txt:[19/Aug/2016:15:47:23 -0400] conn=13 
op=126737 MOD dn="idnsname=rc.usf.edu,cn=dns,dc=rc,dc=usf,dc=edu"
/tmp/adm3-logs-del39-errors.txt:[19/Aug/2016:15:47:26 -0400] conn=13 
op=126758 MOD dn="idnsname=rc.usf.edu,cn=dns,dc=rc,dc=usf,dc=edu"
/tmp/adm3-logs-del39-errors.txt:[19/Aug/2016:15:47:29 -0400] conn=13 
op=126801 MOD dn="idnsname=rc.usf.edu,cn=dns,dc=rc,dc=usf,dc=edu"




On 08/19/2016 10:00 PM, John Desantis wrote:

Ludwig,


you still only grep the replication connection, but before being replicated
the entry has to be added by some client connection, can you get all
references to the entry ?
the log snippet you provide shows also csns with tag=103, which indicate a
MOD, are these MODs for the added entries ? or other mods ?

.

I can't believe I did that!

Ok, so the logs have been rotated (I didn't think to adjust
logrotate..), so there aren't any logs to peruse for the case I've
presented so far.  However, I was able to reproduce the errors by
"bulk" deleting 39 DNS entries, and only the MASTER reported
"replica_generate_next_csn" entries.

Given the size of the logs, I think it would be pointless to do any
kind of sanitization.  I'll go ahead and gzip them for you and email
you off-list.

I've labeled them as MASTER and REPLICA.

John DeSantis


2016-08-19 9:18 GMT-04:00 Ludwig Krispenz <lkris...@redhat.com>:

On 08/18/2016 05:28 PM, John Desantis wrote:

Ludwig,


unfortunately this is not enough to determine what is going on. The
intersting generated/used csn is only logged in the
corresponding RESULT message and these are only the replication
connections,
it would be necessary to see the
original ADD operation, was it added once or twice by a client ?
you could pic

Re: [Freeipa-users] replica_generate_next_csn messages in dirsrv error logs

[Ludwig Krispenz]

0:49 -0400] conn=1395 op=4159
RESULT err=0 tag=103 nentries=0 etime=0
access.20160817-111940:[17/Aug/2016:13:50:49 -0400] conn=1395 op=4160
RESULT err=0 tag=103 nentries=0 etime=0 csn=57b4a4c30016

I'm positive that I was the only one performing DNS updates during
this time, and I was only using 1 console.

Thanks,
John DeSantis


2016-08-18 10:09 GMT-04:00 Ludwig Krispenz <lkris...@redhat.com>:

On 08/18/2016 03:15 PM, John Desantis wrote:

Ludwig,

Thank you for your response!


This is a normal scenario, but you could check if the simultaneous
updates
on 4 and 16 are intentional.

In regards to the simultaneous updates, the only items I have noted so far
are:

*  The time sync between the master (4) and replica (16) was off by
about 1-2 seconds, with the latter being ahead;

yes, this happens, but the replication protocol tries to handle this, in a
replication session the supplier and consumer
exchange their ruvs and if the time differs the csn state generator is
updated with a local or remote offset so that the
generated time is always based on the most advanced clock - on all servers.
And even if you adjust the system time, the csn
time will never go back.

*  There are continual log entries referencing
"replication-multimaster-extop" and "Netscape Replication End Session"
in the dirsrv "access" logs, and during one of the manifestations of
"replica_generate_next_csn", I found this:

PROD:08:46:08-root@REPLICA:/var/log/dirsrv/slapd-DOM-DOM-DOM
# grep -E '17/Aug/2016:13:50:4.*conn=602.*ADD' access.2016081*
access.20160817-124811:[17/Aug/2016:13:50:42 -0400] conn=602 op=4143
ADD
dn="idnsname=server-6-3-sp,idnsname=dom.dom.dom,cn=dns,dc=dom,dc=dom,dc=dom"
access.20160817-124811:[17/Aug/2016:13:50:47 -0400] conn=602 op=4148
ADD
dn="idnsname=server-6-4-sp,idnsname=dom.dom.dom,cn=dns,dc=dom,dc=dom,dc=dom"
access.20160817-124811:[17/Aug/2016:13:50:49 -0400] conn=602 op=4151
ADD
dn="idnsname=server-6-5-sp,idnsname=dom.dom.dom,cn=dns,dc=dom,dc=dom,dc=dom"

PROD:08:47:44-root@MASTER:/var/log/dirsrv/slapd-DOM-DOM-DOM
# grep -E '17/Aug/2016:13:50:4.*conn=1395.*ADD' access.2016081*
access.20160817-111940:[17/Aug/2016:13:50:43 -0400] conn=1395 op=4151
ADD
dn="idnsname=server-6-3-sp,idnsname=dom.dom.dom,cn=dns,dc=dom,dc=dom,dc=dom"
access.20160817-111940:[17/Aug/2016:13:50:49 -0400] conn=1395 op=4158
ADD
dn="idnsname=server-6-5-sp,idnsname=dom.dom.dom,cn=dns,dc=dom,dc=dom,dc=dom"

It looks like the entries for server-6-3-sp and 6-5-sp were referenced
twice.  Do you think that the time being off by 1-2 seconds between
the master and replica could be the issue?  The connection 602 is the
replication between the replica and master, and the connection 1395 is
the replication between the master and replica.

unfortunately this is not enough to determine what is going on. The
intersting generated/used csn is only logged in the
corresponding RESULT message and these are only the replication connections,
it would be necessary to see the
original ADD operation, was it added once or twice by a client ?
you could pick one entry eg server-6-3-sp and grep for all references in the
access logs of both servers  (maybe there are mods as well) and then
get also get the RESULT line for the ops found


Since I know these operations were performed using the console via a
for loop 'ipa dnsrecord-add dom.dom.dom server-6-$i-sp
--a-rec=10.250.12.$i' on one of our login nodes, do you think that
specifying an _srv_ record in the DOMAIN configuration with the
address of the master server, e.g.: ipa_server = _srv_,
MASTER.dom.dom.dom could be the issue (coupled with the time syncing)?

I know that these questions are probably leaning more towards the
389ds team, so feel free to pass me over to them if need be.

I think I can address the ds related questions, but I don't know about
console and dns to assess if the behaviour is normal


Again, thank you very much for responding!

John DeSantis

2016-08-18 4:14 GMT-04:00 Ludwig Krispenz <lkris...@redhat.com>:

On 08/17/2016 08:54 PM, John Desantis wrote:

Hello all,

We've been re-using old host names and IP addresses for a new
deployment of nodes, and recently I've been seeing the messages pasted
below in the slapd-DC.DC.DC "error" log on our nodes.

[17/Aug/2016:10:30:30 -0400] - replica_generate_next_csn:
opcsn=57b475cd00120004 <= basecsn=57b475cf0016, adjusted
opcsn=57b475cf00010004
[17/Aug/2016:11:09:44 -0400] - replica_generate_next_csn:
opcsn=57b47f020004 <= basecsn=57b47f020016, adjusted
opcsn=57b47f030004
[17/Aug/2016:11:09:44 -0400] - replica_generate_next_csn:
opcsn=57b47f040004 <= basecsn=57b47f040016, adjusted
opcsn=57b47f050004
[17/Aug/2016:11:10:33 -0400] - replica_generate_next_csn:
opcsn=57b47f2f0014 <= basecsn=57b47f320016, adjusted
opcsn=57b47f330004
[17/Aug

Re: [Freeipa-users] replica_generate_next_csn messages in dirsrv error logs

[Ludwig Krispenz]

On 08/18/2016 03:15 PM, John Desantis wrote:

Ludwig,

Thank you for your response!


This is a normal scenario, but you could check if the simultaneous updates
on 4 and 16 are intentional.

In regards to the simultaneous updates, the only items I have noted so far are:

*  The time sync between the master (4) and replica (16) was off by
about 1-2 seconds, with the latter being ahead;
yes, this happens, but the replication protocol tries to handle this, in 
a replication session the supplier and consumer
exchange their ruvs and if the time differs the csn state generator is 
updated with a local or remote offset so that the
generated time is always based on the most advanced clock - on all 
servers. And even if you adjust the system time, the csn

time will never go back.

*  There are continual log entries referencing
"replication-multimaster-extop" and "Netscape Replication End Session"
in the dirsrv "access" logs, and during one of the manifestations of
"replica_generate_next_csn", I found this:

PROD:08:46:08-root@REPLICA:/var/log/dirsrv/slapd-DOM-DOM-DOM
# grep -E '17/Aug/2016:13:50:4.*conn=602.*ADD' access.2016081*
access.20160817-124811:[17/Aug/2016:13:50:42 -0400] conn=602 op=4143
ADD dn="idnsname=server-6-3-sp,idnsname=dom.dom.dom,cn=dns,dc=dom,dc=dom,dc=dom"
access.20160817-124811:[17/Aug/2016:13:50:47 -0400] conn=602 op=4148
ADD dn="idnsname=server-6-4-sp,idnsname=dom.dom.dom,cn=dns,dc=dom,dc=dom,dc=dom"
access.20160817-124811:[17/Aug/2016:13:50:49 -0400] conn=602 op=4151
ADD dn="idnsname=server-6-5-sp,idnsname=dom.dom.dom,cn=dns,dc=dom,dc=dom,dc=dom"

PROD:08:47:44-root@MASTER:/var/log/dirsrv/slapd-DOM-DOM-DOM
# grep -E '17/Aug/2016:13:50:4.*conn=1395.*ADD' access.2016081*
access.20160817-111940:[17/Aug/2016:13:50:43 -0400] conn=1395 op=4151
ADD dn="idnsname=server-6-3-sp,idnsname=dom.dom.dom,cn=dns,dc=dom,dc=dom,dc=dom"
access.20160817-111940:[17/Aug/2016:13:50:49 -0400] conn=1395 op=4158
ADD dn="idnsname=server-6-5-sp,idnsname=dom.dom.dom,cn=dns,dc=dom,dc=dom,dc=dom"

It looks like the entries for server-6-3-sp and 6-5-sp were referenced
twice.  Do you think that the time being off by 1-2 seconds between
the master and replica could be the issue?  The connection 602 is the
replication between the replica and master, and the connection 1395 is
the replication between the master and replica.
unfortunately this is not enough to determine what is going on. The 
intersting generated/used csn is only logged in the
corresponding RESULT message and these are only the replication 
connections, it would be necessary to see the

original ADD operation, was it added once or twice by a client ?
you could pick one entry eg server-6-3-sp and grep for all references in 
the access logs of both servers  (maybe there are mods as well) and then

get also get the RESULT line for the ops found


Since I know these operations were performed using the console via a
for loop 'ipa dnsrecord-add dom.dom.dom server-6-$i-sp
--a-rec=10.250.12.$i' on one of our login nodes, do you think that
specifying an _srv_ record in the DOMAIN configuration with the
address of the master server, e.g.: ipa_server = _srv_,
MASTER.dom.dom.dom could be the issue (coupled with the time syncing)?

I know that these questions are probably leaning more towards the
389ds team, so feel free to pass me over to them if need be.
I think I can address the ds related questions, but I don't know about 
console and dns to assess if the behaviour is normal


Again, thank you very much for responding!

John DeSantis

2016-08-18 4:14 GMT-04:00 Ludwig Krispenz <lkris...@redhat.com>:

On 08/17/2016 08:54 PM, John Desantis wrote:

Hello all,

We've been re-using old host names and IP addresses for a new
deployment of nodes, and recently I've been seeing the messages pasted
below in the slapd-DC.DC.DC "error" log on our nodes.

[17/Aug/2016:10:30:30 -0400] - replica_generate_next_csn:
opcsn=57b475cd00120004 <= basecsn=57b475cf0016, adjusted
opcsn=57b475cf00010004
[17/Aug/2016:11:09:44 -0400] - replica_generate_next_csn:
opcsn=57b47f020004 <= basecsn=57b47f020016, adjusted
opcsn=57b47f030004
[17/Aug/2016:11:09:44 -0400] - replica_generate_next_csn:
opcsn=57b47f040004 <= basecsn=57b47f040016, adjusted
opcsn=57b47f050004
[17/Aug/2016:11:10:33 -0400] - replica_generate_next_csn:
opcsn=57b47f2f0014 <= basecsn=57b47f320016, adjusted
opcsn=57b47f330004
[17/Aug/2016:13:50:45 -0400] - replica_generate_next_csn:
opcsn=57b4a4bb00090004 <= basecsn=57b4a4bc0016, adjusted
opcsn=57b4a4bc00010004
[17/Aug/2016:13:52:54 -0400] - replica_generate_next_csn:
opcsn=57b4a53e000a0004 <= basecsn=57b4a53f0016, adjusted
opcsn=57b4a53f00010004
[17/Aug/2016:13:53:15 -0400] - replica_generate_next_csn:
opcsn=57b4a55200070004 <= ba

Re: [Freeipa-users] replica_generate_next_csn messages in dirsrv error logs

[Ludwig Krispenz]

On 08/17/2016 08:54 PM, John Desantis wrote:

Hello all,

We've been re-using old host names and IP addresses for a new
deployment of nodes, and recently I've been seeing the messages pasted
below in the slapd-DC.DC.DC "error" log on our nodes.

[17/Aug/2016:10:30:30 -0400] - replica_generate_next_csn:
opcsn=57b475cd00120004 <= basecsn=57b475cf0016, adjusted
opcsn=57b475cf00010004
[17/Aug/2016:11:09:44 -0400] - replica_generate_next_csn:
opcsn=57b47f020004 <= basecsn=57b47f020016, adjusted
opcsn=57b47f030004
[17/Aug/2016:11:09:44 -0400] - replica_generate_next_csn:
opcsn=57b47f040004 <= basecsn=57b47f040016, adjusted
opcsn=57b47f050004
[17/Aug/2016:11:10:33 -0400] - replica_generate_next_csn:
opcsn=57b47f2f0014 <= basecsn=57b47f320016, adjusted
opcsn=57b47f330004
[17/Aug/2016:13:50:45 -0400] - replica_generate_next_csn:
opcsn=57b4a4bb00090004 <= basecsn=57b4a4bc0016, adjusted
opcsn=57b4a4bc00010004
[17/Aug/2016:13:52:54 -0400] - replica_generate_next_csn:
opcsn=57b4a53e000a0004 <= basecsn=57b4a53f0016, adjusted
opcsn=57b4a53f00010004
[17/Aug/2016:13:53:15 -0400] - replica_generate_next_csn:
opcsn=57b4a55200070004 <= basecsn=57b4a5530016, adjusted
opcsn=57b4a55300010004
[17/Aug/2016:13:53:32 -0400] - replica_generate_next_csn:
opcsn=57b4a56200090004 <= basecsn=57b4a5640016, adjusted
opcsn=57b4a56400010004
Each modification (add/del/mod) gets a csn assignged used in replication 
update resolution. And each assigned csn has to newer than an existing 
one. The messages you see is from code that double checks that the entry 
doesn't have already a lareg csn - and adjusts it.
The logs indicate that entries are more or less concurrently updated on 
replica 4 and 16, and the updates from16 are received while processing 
the updates on 4.
This is a normal scenario, but you could check if the simultaneous 
updates on 4 and 16 are intentional.


They seem to only occur when updating DNS entries, whether on the
console or via the GUI (tail -f'ing the log).

A search in this mailing-list returns nothing, but a message is found
on the 389-ds list [1];  it seems to suggest that the messages aren't
fatal and are purely informational, yet if they are occurring
constantly that there could be a problem with the replication
algorithm and/or deployment.

We're using ipa-server 3.0.0-47 and 389-ds 1.2.11.15-60.  Nothing has
changed on the deployment side of things, and I don't recall seeing
this message before.

I'm wondering if it's safe to disregard these messages due to the
re-use of the entries, or if something else should be looked into.

Thank you,
John DeSantis

[1] https://fedorahosted.org/389/ticket/47959



--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Problem with replication

[Ludwig Krispenz]

On 08/12/2016 04:10 PM, Louis Francoeur wrote:


Since the rpm update to 
ipa-server-dns-4.2.0-15.0.1.el7.centos.18.x86_64 (running on Centos 7),



most of my replication started to failed with:

what do you mean by "most of", if some servers still work and others 
don't is there something different ?



last update status: -1 Incremental update has failed and requires 
administrator actionLDAP error: Can't contact LDAP server


what is in the error log of directory server ? Identify one broken 
replication connection and check both supplier and consumer side



Then setup contains about 10 ipa servers in 5 different locations.


But i went and ran an ipa-replica-conncheck i get this:


# ipa-replica-conncheck --replica server.domain.local
Check connection from master to remote replica 'server.domain.local':
   Directory Service: Unsecure port (389): OK
   Directory Service: Secure port (636): OK
   Kerberos KDC: TCP (88): OK
   Kerberos KDC: UDP (88): WARNING
   Kerberos Kpasswd: TCP (464): OK
   Kerberos Kpasswd: UDP (464): WARNING
   HTTP Server: Unsecure port (80): OK
   HTTP Server: Secure port (443): OK
The following UDP ports could not be verified as open: 88, 464
This can happen if they are already bound to an application
and ipa-replica-conncheck cannot attach own UDP responder.

Connection from master to replica is OK.



I even ran the following without issue:

# kinit -kt /etc/dirsrv/ds.keytab ldap/`hostname`
# klist
# ldapsearch -Y GSSAPI -h `hostname` -b "" -s base
# ldapsearch -Y GSSAPI -h the.other.master.fqdn -b "" -s base

Not really sure what to check for next?

Any hint?


Thanks

Louis Francoeur





--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Could not delete change record

[Ludwig Krispenz]

On 07/12/2016 11:25 AM, Christophe TREFOIS wrote:

Hi,

I have 3 replicas running 4.1 and 3 replicas running 4.2.

One of the 4.2 replicas is the new master (CRL) and is at the moment 
replicating against the old 4.1 cluster (we are in the process of 
migrating).


Upon restart of the 4.2 master, I receive many messages in slapd error 
log about delete_changerecord as seen below.


Is this something to worry about, or will it go away by itself?
it should go away, it is a problem of incorrect starting point for retro 
changelog trimming and it tries to remove already deleted records.


Thank you for your help,

[12/Jul/2016:11:16:43 +0200] DSRetroclPlugin - delete_changerecord: 
could not delete change record 15892 (rc: 32)
[12/Jul/2016:11:16:43 +0200] DSRetroclPlugin - delete_changerecord: 
could not delete change record 15893 (rc: 32)
[12/Jul/2016:11:16:43 +0200] DSRetroclPlugin - delete_changerecord: 
could not delete change record 15894 (rc: 32)
[12/Jul/2016:11:16:43 +0200] DSRetroclPlugin - delete_changerecord: 
could not delete change record 15895 (rc: 32)
[12/Jul/2016:11:16:43 +0200] DSRetroclPlugin - delete_changerecord: 
could not delete change record 15896 (rc: 32)
[12/Jul/2016:11:16:43 +0200] DSRetroclPlugin - delete_changerecord: 
could not delete change record 15897 (rc: 32)
[12/Jul/2016:11:16:43 +0200] DSRetroclPlugin - delete_changerecord: 
could not delete change record 15898 (rc: 32)
[12/Jul/2016:11:16:43 +0200] DSRetroclPlugin - delete_changerecord: 
could not delete change record 15899 (rc: 32)
[12/Jul/2016:11:16:43 +0200] DSRetroclPlugin - delete_changerecord: 
could not delete change record 15900 (rc: 32)
[12/Jul/2016:11:16:43 +0200] DSRetroclPlugin - delete_changerecord: 
could not delete change record 15901 (rc: 32)


*Christophe*





--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] FreeIPA (directory service) Crash several times a day

[Ludwig Krispenz]

On 07/05/2016 12:08 PM, Omar AKHAM wrote:

OK thanks. Ticket URL : https://fedorahosted.org/freeipa/ticket/6030
thanks, I tried to reproduce and failed so far, could you add some 
information to the ticket on

- how the entry was created
- a full entry which was seen to crash the server, you don't need to 
reveal any real data, jsur which objectclasses and attributes the entry has


On 2016-07-05 10:51, Ludwig Krispenz wrote:

well, this does not have more information:
#0  0x7efe7167c4c0 in ipapwd_keyset_free () from
/usr/lib64/dirsrv/plugins/libipa_pwd_extop.so
No symbol table info available.
#1  0x7efe7167c742 in ipapwd_encrypt_encode_key () from
/usr/lib64/dirsrv/plugins/libipa_pwd_extop.so
No symbol table info available.
#2  0x7efe7167c9c8 in ipapwd_gen_hashes () from
/usr/lib64/dirsrv/plugins/libipa_pwd_extop.so
No symbol table info available.
#3  0x7efe7167c0a7 in ipapwd_SetPassword () from
/usr/lib64/dirsrv/plugins/libipa_pwd_extop.so
No symbol table info available.
#4  0x7efe7167e458 in ipapwd_pre_bind () from
/usr/lib64/dirsrv/plugins/libipa_pwd_extop.so
No symbol table info available.

and it looks like a bug in the ipapwd plugin, we would have to
reproduce and work on a fix. I don't see any immediate relief unless
you cannot prevent clients from using password containing arbitrar
octets.
Please open a ticket to get this worked on:
https://fedorahosted.org/freeipa/newticket

Ludwig

On 07/05/2016 12:07 AM, Omar AKHAM wrote:

Ok, here is a new core file : http://pastebin.com/2cJQymHd

Best regards

On 2016-07-04 09:39, Ludwig Krispenz wrote:

On 07/03/2016 03:04 PM, Omar AKHAM wrote:

Where can i find core file of ipa-server?

you still need to look for the core file of slapd, but IPA deploys
plugins for slapd and that  is why you need the debuginfo for
ipa-server for a better analysis of the slapd core.


On 2016-07-01 13:29, Ludwig Krispenz wrote:

please keep the discussion on the mailing list
On 07/01/2016 01:17 PM, Omar AKHAM wrote:

Which package to install ? ipa-debuginfo?

yes


2 other crashes last night, with a different user bind this time :

rawdn = 0x7f620003a200 
"uid=XXX,cn=users,cn=accounts,dc=XXX,dc=XX"
dn = 0x7f62000238b0 
"uid=XXX,cn=users,cn=accounts,dc=XXX,dc=XX"

saslmech = 0x0
cred = {bv_len = 9, bv_val = 0x7f6200034af0 
"nw_PA\250\063\065\067"}

be = 0x7f6254941c20
ber_rc = 
rc = 0
sdn = 0x7f62000313f0
bind_sdn_in_pb = 1
referral = 0x0
errorbuf = '\000' ...
supported = 
pmech = 
authtypebuf = 
"\000\000\000\000\000\000\000\000\370\030\002\000b\177\000\000\360\030\002\000b\177\000\000\320\030\002\000b\177\000\000\001\000
\000\000\000\000\000\000\250\311\377+b\177\000\000\320\352\377+b\177\000\000\200\376\002\000b\177\000\000\262\202\211Rb\177\000\000\260\311\377+b\177\ 
000\000\000\000\000\000\000\000\000\000&\272\200Rb\177\000\000\000\000\000\000\000\000\000\000<\224\204Rb\177\000\000\260\311\377+b\177\000\000\000\00 
0\000\000\000\000\000\000\210\311\377+b\177\000\000\250\311\377+b\177", 
'\000' , "\002\000\000\000 
\305\363Tb\177\000\000\377\377\37
7\377\377\377\377\377\320\030\002\000b\177\000\000\000\000\000\000\000\000\000\000~a\003\000b\177", 
'\000' 

    bind_target_entry = 0x0



On 2016-06-30 18:16, Ludwig Krispenz wrote:

On 06/30/2016 05:54 PM, d...@mdfive.dz wrote:
The crash is random, sometimes the user binds without 
probleme, sometimes it bind and there is the error message of 
ipa plugin without dirsrv crash. But when it crashes, this 
user's bind is found in the new generated core file!

ok, so the user might try or use different passwords. it could be
helpful if you can install the debuginfo for the ipa-server 
package

and get a new stack. Please post it to teh list, you can X the
credentials in the core, although I think they will not be proper
credentials.

Ludwig


On 2016-06-30 14:50, Ludwig Krispenz wrote:

On 06/30/2016 02:45 PM, Ludwig Krispenz wrote:


On 06/30/2016 02:27 PM, d...@mdfive.dz wrote:

Hi,

Please find strace on a core file : 
http://pastebin.com/v9cUzau4

the crash is in an IPA plugin, ipa_pwd_extop,
to get a better stack you would have to install also the 
debuginfo for ipa-server.

but tje stack matches the error messages you have seen
[30/Jun/2016:09:35:19 +0100] ipapwd_encrypt_encode_key - [file
encoding.c, line 171]: generating kerberos keys failed [Invalid
argument]
[30/Jun/2016:09:35:19 +0100] ipapwd_gen_hashes - [file 
encoding.c,

line 225]: key encryption/encoding failed
they are from the function sin the call stack.

Looks like the user has a password with a \351 char:
cred = {bv_len = 15, bv_val = 0x7fc7880013a0 
"d\351sertification"}


does the crash always happen with a bind from this user ?


and then someone familiar with this plugin should look into it


Regards


On 2016-06-30 12:13, Ludwig Krispenz wrot

Re: [Freeipa-users] FreeIPA (directory service) Crash several times a day

[Ludwig Krispenz]

well, this does not have more information:
#0  0x7efe7167c4c0 in ipapwd_keyset_free () from 
/usr/lib64/dirsrv/plugins/libipa_pwd_extop.so

No symbol table info available.
#1  0x7efe7167c742 in ipapwd_encrypt_encode_key () from 
/usr/lib64/dirsrv/plugins/libipa_pwd_extop.so

No symbol table info available.
#2  0x7efe7167c9c8 in ipapwd_gen_hashes () from 
/usr/lib64/dirsrv/plugins/libipa_pwd_extop.so

No symbol table info available.
#3  0x7efe7167c0a7 in ipapwd_SetPassword () from 
/usr/lib64/dirsrv/plugins/libipa_pwd_extop.so

No symbol table info available.
#4  0x7efe7167e458 in ipapwd_pre_bind () from 
/usr/lib64/dirsrv/plugins/libipa_pwd_extop.so

No symbol table info available.

and it looks like a bug in the ipapwd plugin, we would have to reproduce 
and work on a fix. I don't see any immediate relief unless you cannot 
prevent clients from using password containing arbitrar octets.
Please open a ticket to get this worked on: 
https://fedorahosted.org/freeipa/newticket


Ludwig

On 07/05/2016 12:07 AM, Omar AKHAM wrote:

Ok, here is a new core file : http://pastebin.com/2cJQymHd

Best regards

On 2016-07-04 09:39, Ludwig Krispenz wrote:

On 07/03/2016 03:04 PM, Omar AKHAM wrote:

Where can i find core file of ipa-server?

you still need to look for the core file of slapd, but IPA deploys
plugins for slapd and that  is why you need the debuginfo for
ipa-server for a better analysis of the slapd core.


On 2016-07-01 13:29, Ludwig Krispenz wrote:

please keep the discussion on the mailing list
On 07/01/2016 01:17 PM, Omar AKHAM wrote:

Which package to install ? ipa-debuginfo?

yes


2 other crashes last night, with a different user bind this time :

rawdn = 0x7f620003a200 
"uid=XXX,cn=users,cn=accounts,dc=XXX,dc=XX"
dn = 0x7f62000238b0 
"uid=XXX,cn=users,cn=accounts,dc=XXX,dc=XX"

saslmech = 0x0
cred = {bv_len = 9, bv_val = 0x7f6200034af0 
"nw_PA\250\063\065\067"}

be = 0x7f6254941c20
ber_rc = 
rc = 0
sdn = 0x7f62000313f0
bind_sdn_in_pb = 1
referral = 0x0
errorbuf = '\000' ...
supported = 
pmech = 
authtypebuf = 
"\000\000\000\000\000\000\000\000\370\030\002\000b\177\000\000\360\030\002\000b\177\000\000\320\030\002\000b\177\000\000\001\000
\000\000\000\000\000\000\250\311\377+b\177\000\000\320\352\377+b\177\000\000\200\376\002\000b\177\000\000\262\202\211Rb\177\000\000\260\311\377+b\177\ 
000\000\000\000\000\000\000\000\000\000&\272\200Rb\177\000\000\000\000\000\000\000\000\000\000<\224\204Rb\177\000\000\260\311\377+b\177\000\000\000\00 
0\000\000\000\000\000\000\210\311\377+b\177\000\000\250\311\377+b\177", 
'\000' , "\002\000\000\000 
\305\363Tb\177\000\000\377\377\37
7\377\377\377\377\377\320\030\002\000b\177\000\000\000\000\000\000\000\000\000\000~a\003\000b\177", 
'\000' 

    bind_target_entry = 0x0



On 2016-06-30 18:16, Ludwig Krispenz wrote:

On 06/30/2016 05:54 PM, d...@mdfive.dz wrote:
The crash is random, sometimes the user binds without probleme, 
sometimes it bind and there is the error message of ipa plugin 
without dirsrv crash. But when it crashes, this user's bind is 
found in the new generated core file!

ok, so the user might try or use different passwords. it could be
helpful if you can install the debuginfo for the ipa-server package
and get a new stack. Please post it to teh list, you can X the
credentials in the core, although I think they will not be proper
credentials.

Ludwig


On 2016-06-30 14:50, Ludwig Krispenz wrote:

On 06/30/2016 02:45 PM, Ludwig Krispenz wrote:


On 06/30/2016 02:27 PM, d...@mdfive.dz wrote:

Hi,

Please find strace on a core file : http://pastebin.com/v9cUzau4

the crash is in an IPA plugin, ipa_pwd_extop,
to get a better stack you would have to install also the 
debuginfo for ipa-server.

but tje stack matches the error messages you have seen
[30/Jun/2016:09:35:19 +0100] ipapwd_encrypt_encode_key - [file
encoding.c, line 171]: generating kerberos keys failed [Invalid
argument]
[30/Jun/2016:09:35:19 +0100] ipapwd_gen_hashes - [file 
encoding.c,

line 225]: key encryption/encoding failed
they are from the function sin the call stack.

Looks like the user has a password with a \351 char:
cred = {bv_len = 15, bv_val = 0x7fc7880013a0 "d\351sertification"}

does the crash always happen with a bind from this user ?


and then someone familiar with this plugin should look into it


Regards


On 2016-06-30 12:13, Ludwig Krispenz wrote:

can you get a core file ?
http://www.port389.org/docs/389ds/FAQ/faq.html#debug_crashes


On 06/30/2016 11:28 AM, d...@mdfive.dz wrote:

Hi,

The Directory Services crashes several times a day. It's 
installed on CentOS 7 VM :


Installed Packages
Name: ipa-server
Arch: x86_64
Version : 4.2.0

# ipactl status
Directory Service: STOPPED
krb5kdc Service: RUNNING
kadmin Service: RUNNING
ipa_me

Re: [Freeipa-users] FreeIPA (directory service) Crash several times a day

[Ludwig Krispenz]

On 07/03/2016 03:04 PM, Omar AKHAM wrote:

Where can i find core file of ipa-server?
you still need to look for the core file of slapd, but IPA deploys 
plugins for slapd and that  is why you need the debuginfo for ipa-server 
for a better analysis of the slapd core.


On 2016-07-01 13:29, Ludwig Krispenz wrote:

please keep the discussion on the mailing list
On 07/01/2016 01:17 PM, Omar AKHAM wrote:

Which package to install ? ipa-debuginfo?

yes


2 other crashes last night, with a different user bind this time :

rawdn = 0x7f620003a200 
"uid=XXX,cn=users,cn=accounts,dc=XXX,dc=XX"

dn = 0x7f62000238b0 "uid=XXX,cn=users,cn=accounts,dc=XXX,dc=XX"
saslmech = 0x0
cred = {bv_len = 9, bv_val = 0x7f6200034af0 
"nw_PA\250\063\065\067"}

be = 0x7f6254941c20
ber_rc = 
rc = 0
sdn = 0x7f62000313f0
bind_sdn_in_pb = 1
referral = 0x0
errorbuf = '\000' ...
supported = 
pmech = 
authtypebuf = 
"\000\000\000\000\000\000\000\000\370\030\002\000b\177\000\000\360\030\002\000b\177\000\000\320\030\002\000b\177\000\000\001\000
\000\000\000\000\000\000\250\311\377+b\177\000\000\320\352\377+b\177\000\000\200\376\002\000b\177\000\000\262\202\211Rb\177\000\000\260\311\377+b\177\ 
000\000\000\000\000\000\000\000\000\000&\272\200Rb\177\000\000\000\000\000\000\000\000\000\000<\224\204Rb\177\000\000\260\311\377+b\177\000\000\000\00 
0\000\000\000\000\000\000\210\311\377+b\177\000\000\250\311\377+b\177", 
'\000' , "\002\000\000\000 
\305\363Tb\177\000\000\377\377\37
7\377\377\377\377\377\320\030\002\000b\177\000\000\000\000\000\000\000\000\000\000~a\003\000b\177", 
'\000' 

    bind_target_entry = 0x0



On 2016-06-30 18:16, Ludwig Krispenz wrote:

On 06/30/2016 05:54 PM, d...@mdfive.dz wrote:
The crash is random, sometimes the user binds without probleme, 
sometimes it bind and there is the error message of ipa plugin 
without dirsrv crash. But when it crashes, this user's bind is 
found in the new generated core file!

ok, so the user might try or use different passwords. it could be
helpful if you can install the debuginfo for the ipa-server package
and get a new stack. Please post it to teh list, you can X the
credentials in the core, although I think they will not be proper
credentials.

Ludwig


On 2016-06-30 14:50, Ludwig Krispenz wrote:

On 06/30/2016 02:45 PM, Ludwig Krispenz wrote:


On 06/30/2016 02:27 PM, d...@mdfive.dz wrote:

Hi,

Please find strace on a core file : http://pastebin.com/v9cUzau4

the crash is in an IPA plugin, ipa_pwd_extop,
to get a better stack you would have to install also the 
debuginfo for ipa-server.

but tje stack matches the error messages you have seen
[30/Jun/2016:09:35:19 +0100] ipapwd_encrypt_encode_key - [file
encoding.c, line 171]: generating kerberos keys failed [Invalid
argument]
[30/Jun/2016:09:35:19 +0100] ipapwd_gen_hashes - [file 
encoding.c,

line 225]: key encryption/encoding failed
they are from the function sin the call stack.

Looks like the user has a password with a \351 char:
cred = {bv_len = 15, bv_val = 0x7fc7880013a0 "d\351sertification"}

does the crash always happen with a bind from this user ?


and then someone familiar with this plugin should look into it


Regards


On 2016-06-30 12:13, Ludwig Krispenz wrote:

can you get a core file ?
http://www.port389.org/docs/389ds/FAQ/faq.html#debug_crashes


On 06/30/2016 11:28 AM, d...@mdfive.dz wrote:

Hi,

The Directory Services crashes several times a day. It's 
installed on CentOS 7 VM :


Installed Packages
Name: ipa-server
Arch: x86_64
Version : 4.2.0

# ipactl status
Directory Service: STOPPED
krb5kdc Service: RUNNING
kadmin Service: RUNNING
ipa_memcached Service: RUNNING
httpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa: INFO: The ipactl command was successful


Before each crash, I have these messages in 
/var/log/dirsrv/slapd-X/errors :


[30/Jun/2016:09:35:19 +0100] ipapwd_encrypt_encode_key - 
[file encoding.c, line 171]: generating kerberos keys failed 
[Invalid argument]
[30/Jun/2016:09:35:19 +0100] ipapwd_gen_hashes - [file 
encoding.c, line 225]: key encryption/encoding failed



Any help?
Best regards



-- Red Hat GmbH, http://www.de.redhat.com/, Registered seat: 
Grasbrunn,

Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael
O'Neill, Eric Shander




-- Red Hat GmbH, http://www.de.redhat.com/, Registered seat: 
Grasbrunn,

Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael
O'Neill, Eric Shander


--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

--

Re: [Freeipa-users] FreeIPA (directory service) Crash several times a day

[Ludwig Krispenz]

please keep the discussion on the mailing list
On 07/01/2016 01:17 PM, Omar AKHAM wrote:

Which package to install ? ipa-debuginfo?

yes


2 other crashes last night, with a different user bind this time :

rawdn = 0x7f620003a200 
"uid=XXX,cn=users,cn=accounts,dc=XXX,dc=XX"

dn = 0x7f62000238b0 "uid=XXX,cn=users,cn=accounts,dc=XXX,dc=XX"
saslmech = 0x0
cred = {bv_len = 9, bv_val = 0x7f6200034af0 
"nw_PA\250\063\065\067"}

be = 0x7f6254941c20
ber_rc = 
rc = 0
sdn = 0x7f62000313f0
bind_sdn_in_pb = 1
referral = 0x0
errorbuf = '\000' ...
supported = 
pmech = 
authtypebuf = 
"\000\000\000\000\000\000\000\000\370\030\002\000b\177\000\000\360\030\002\000b\177\000\000\320\030\002\000b\177\000\000\001\000
\000\000\000\000\000\000\250\311\377+b\177\000\000\320\352\377+b\177\000\000\200\376\002\000b\177\000\000\262\202\211Rb\177\000\000\260\311\377+b\177\ 

000\000\000\000\000\000\000\000\000\000&\272\200Rb\177\000\000\000\000\000\000\000\000\000\000<\224\204Rb\177\000\000\260\311\377+b\177\000\000\000\00 

0\000\000\000\000\000\000\210\311\377+b\177\000\000\250\311\377+b\177", '\000' 
, "\002\000\000\000 \305\363Tb\177\000\000\377\377\37
7\377\377\377\377\377\320\030\002\000b\177\000\000\000\000\000\000\000\000\000\000~a\003\000b\177", 
'\000' 

    bind_target_entry = 0x0



On 2016-06-30 18:16, Ludwig Krispenz wrote:

On 06/30/2016 05:54 PM, d...@mdfive.dz wrote:
The crash is random, sometimes the user binds without probleme, 
sometimes it bind and there is the error message of ipa plugin 
without dirsrv crash. But when it crashes, this user's bind is found 
in the new  generated core file!

ok, so the user might try or use different passwords. it could be
helpful if you can install the debuginfo for the ipa-server package
and get a new stack. Please post it to teh list, you can X the
credentials in the core, although I think they will not be proper
credentials.

Ludwig


On 2016-06-30 14:50, Ludwig Krispenz wrote:

On 06/30/2016 02:45 PM, Ludwig Krispenz wrote:


On 06/30/2016 02:27 PM, d...@mdfive.dz wrote:

Hi,

Please find strace on a core file : http://pastebin.com/v9cUzau4

the crash is in an IPA plugin, ipa_pwd_extop,
to get a better stack you would have to install also the debuginfo 
for ipa-server.

but tje stack matches the error messages you have seen
[30/Jun/2016:09:35:19 +0100] ipapwd_encrypt_encode_key - [file
encoding.c, line 171]: generating kerberos keys failed [Invalid
argument]
[30/Jun/2016:09:35:19 +0100] ipapwd_gen_hashes - [file encoding.c,
line 225]: key encryption/encoding failed
they are from the function sin the call stack.

Looks like the user has a password with a \351 char:
cred = {bv_len = 15, bv_val = 0x7fc7880013a0 "d\351sertification"}

does the crash always happen with a bind from this user ?


and then someone familiar with this plugin should look into it


Regards


On 2016-06-30 12:13, Ludwig Krispenz wrote:

can you get a core file ?
http://www.port389.org/docs/389ds/FAQ/faq.html#debug_crashes


On 06/30/2016 11:28 AM, d...@mdfive.dz wrote:

Hi,

The Directory Services crashes several times a day. It's 
installed on CentOS 7 VM :


Installed Packages
Name: ipa-server
Arch: x86_64
Version : 4.2.0

# ipactl status
Directory Service: STOPPED
krb5kdc Service: RUNNING
kadmin Service: RUNNING
ipa_memcached Service: RUNNING
httpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa: INFO: The ipactl command was successful


Before each crash, I have these messages in 
/var/log/dirsrv/slapd-X/errors :


[30/Jun/2016:09:35:19 +0100] ipapwd_encrypt_encode_key - 
[file encoding.c, line 171]: generating kerberos keys failed 
[Invalid argument]
[30/Jun/2016:09:35:19 +0100] ipapwd_gen_hashes - [file 
encoding.c, line 225]: key encryption/encoding failed



Any help?
Best regards



-- Red Hat GmbH, http://www.de.redhat.com/, Registered seat: 
Grasbrunn,

Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael
O'Neill, Eric Shander




-- Red Hat GmbH, http://www.de.redhat.com/, Registered seat: 
Grasbrunn,

Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael
O'Neill, Eric Shander


--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] FreeIPA (directory service) Crash several times a day

[Ludwig Krispenz]

On 06/30/2016 02:45 PM, Ludwig Krispenz wrote:


On 06/30/2016 02:27 PM, d...@mdfive.dz wrote:

Hi,

Please find strace on a core file : http://pastebin.com/v9cUzau4

the crash is in an IPA plugin, ipa_pwd_extop,
to get a better stack you would have to install also the debuginfo for 
ipa-server.

but tje stack matches the error messages you have seen
[30/Jun/2016:09:35:19 +0100] ipapwd_encrypt_encode_key - [file 
encoding.c, line 171]: generating kerberos keys failed [Invalid argument]
[30/Jun/2016:09:35:19 +0100] ipapwd_gen_hashes - [file encoding.c, 
line 225]: key encryption/encoding failed

they are from the function sin the call stack.

Looks like the user has a password with a \351 char:
cred = {bv_len = 15, bv_val = 0x7fc7880013a0 "d\351sertification"}

does the crash always happen with a bind from this user ?


and then someone familiar with this plugin should look into it


Regards


On 2016-06-30 12:13, Ludwig Krispenz wrote:

can you get a core file ?
http://www.port389.org/docs/389ds/FAQ/faq.html#debug_crashes


On 06/30/2016 11:28 AM, d...@mdfive.dz wrote:

Hi,

The Directory Services crashes several times a day. It's installed 
on CentOS 7 VM :


Installed Packages
Name: ipa-server
Arch: x86_64
Version : 4.2.0

# ipactl status
Directory Service: STOPPED
krb5kdc Service: RUNNING
kadmin Service: RUNNING
ipa_memcached Service: RUNNING
httpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa: INFO: The ipactl command was successful


Before each crash, I have these messages in 
/var/log/dirsrv/slapd-X/errors :


[30/Jun/2016:09:35:19 +0100] ipapwd_encrypt_encode_key - [file 
encoding.c, line 171]: generating kerberos keys failed [Invalid 
argument]
[30/Jun/2016:09:35:19 +0100] ipapwd_gen_hashes - [file 
encoding.c, line 225]: key encryption/encoding failed



Any help?
Best regards



--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael
O'Neill, Eric Shander




--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] FreeIPA (directory service) Crash several times a day

[Ludwig Krispenz]

On 06/30/2016 02:27 PM, d...@mdfive.dz wrote:

Hi,

Please find strace on a core file : http://pastebin.com/v9cUzau4

the crash is in an IPA plugin, ipa_pwd_extop,
to get a better stack you would have to install also the debuginfo for 
ipa-server.

and then someone familiar with this plugin should look into it


Regards


On 2016-06-30 12:13, Ludwig Krispenz wrote:

can you get a core file ?
http://www.port389.org/docs/389ds/FAQ/faq.html#debug_crashes


On 06/30/2016 11:28 AM, d...@mdfive.dz wrote:

Hi,

The Directory Services crashes several times a day. It's installed 
on CentOS 7 VM :


Installed Packages
Name: ipa-server
Arch: x86_64
Version : 4.2.0

# ipactl status
Directory Service: STOPPED
krb5kdc Service: RUNNING
kadmin Service: RUNNING
ipa_memcached Service: RUNNING
httpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa: INFO: The ipactl command was successful


Before each crash, I have these messages in 
/var/log/dirsrv/slapd-X/errors :


[30/Jun/2016:09:35:19 +0100] ipapwd_encrypt_encode_key - [file 
encoding.c, line 171]: generating kerberos keys failed [Invalid 
argument]
[30/Jun/2016:09:35:19 +0100] ipapwd_gen_hashes - [file 
encoding.c, line 225]: key encryption/encoding failed



Any help?
Best regards



--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael
O'Neill, Eric Shander


--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] FreeIPA (directory service) Crash several times a day

[Ludwig Krispenz]

can you get a core file ?
http://www.port389.org/docs/389ds/FAQ/faq.html#debug_crashes


On 06/30/2016 11:28 AM, d...@mdfive.dz wrote:

Hi,

The Directory Services crashes several times a day. It's installed on 
CentOS 7 VM :


Installed Packages
Name: ipa-server
Arch: x86_64
Version : 4.2.0

# ipactl status
Directory Service: STOPPED
krb5kdc Service: RUNNING
kadmin Service: RUNNING
ipa_memcached Service: RUNNING
httpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa: INFO: The ipactl command was successful


Before each crash, I have these messages in 
/var/log/dirsrv/slapd-X/errors :


[30/Jun/2016:09:35:19 +0100] ipapwd_encrypt_encode_key - [file 
encoding.c, line 171]: generating kerberos keys failed [Invalid argument]
[30/Jun/2016:09:35:19 +0100] ipapwd_gen_hashes - [file encoding.c, 
line 225]: key encryption/encoding failed



Any help?
Best regards



--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] multiple ds instances (maybe off-topic)

[Ludwig Krispenz]

On 06/28/2016 10:33 AM, Natxo Asenjo wrote:


hi Ludwig,

On Tue, Jun 28, 2016 at 10:03 AM, Ludwig Krispenz <lkris...@redhat.com 
<mailto:lkris...@redhat.com>> wrote:



On 06/28/2016 09:50 AM, Natxo Asenjo wrote:


I'd like to have internally all sort of ldap access, but
externally onlly certificate based, for example.

If there is a way to do that know that I am not aware of I'd be
very interested to know it as well ;-). Right now we solve this
problems using vpn connections with third parties, but ideally
one could just open the port to the internet if only that kind of
access was allowed.

maybe you can achieve this with access control, there are all kind
of rules to allow access based on client's ip address, domain,
security strength, authentication method - and combinations of them.


Do you mean something like explained here: 
http://directory.fedoraproject.org/docs/389ds/design/rootdn-access-control.html 
?

I was thinking of something like this (and the other bind rules):

https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/10/html/Administration_Guide/Managing_Access_Control-Bind_Rules.html#Bind_Rules-Defining_Access_Based_on_Authentication_Method

the link you sent is about restraing access of directory manager, which 
is not subject to normal acis


Thanks!
--
Groeten,
natxo




--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] multiple ds instances (maybe off-topic)

[Ludwig Krispenz]

On 06/28/2016 09:50 AM, Natxo Asenjo wrote:



On Tue, Jun 28, 2016 at 9:07 AM, Alexander Bokovoy 
> wrote:


On Tue, 28 Jun 2016, Natxo Asenjo wrote:

hi,

according to the RHDS documentation (

https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/8.1/html-single/Using_the_Admin_Server/index.html)
one can have multiple directory server instances on the same hosts

Would it be interesting to offer this functionality in
freeipa.org ? The
business case would be to allow different kinds of
authentication per
instance/port. So one could block standard ldap connections on
port 389 to
the internet, for instance, but allow them on another port
only if using
external/GSSAPI auth, so no passswords would be involved.

This is not how instances work in 389-ds. Each instance is fully
independent of another one, including database content and structure.
You cannot have instance that shares the same content with another one
unless you enable database chaining (and then there are some
limitations).


ok, thanks for the info.

We used to have CA instance separate from the main IPA instance, for
example, but then merged them together in the same instance using two
different backends.

Standard IPA 389-ds instance already allows its access on the unix
domain
socket with EXTERNAL/GSSAPI authentication. It is visible only within
the scope of the IPA master host, of course.

I'm still not sure what exactly you would like to achieve. All ports
that 389-ds listens to do support the same authentication methods
except
LDAPI protocol (unix domain sockets) which supports automapping
between
POSIX ID and a user object that it maps to.


I'd like to have internally all sort of ldap access, but externally 
onlly certificate based, for example.


If there is a way to do that know that I am not aware of I'd be very 
interested to know it as well ;-). Right now we solve this problems 
using vpn connections with third parties, but ideally one could just 
open the port to the internet if only that kind of access was allowed.
maybe you can achieve this with access control, there are all kind of 
rules to allow access based on client's ip address, domain, security 
strength, authentication method - and combinations of them.



Thanks for your time.

--
regards,
Natxo





--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] replication - ruv errors

[Ludwig Krispenz]

On 06/07/2016 06:17 PM, Andy Brittingham wrote:

Hello,

I'm having issues with freeipa replication. Currently we have 4 
Freeipa servers, in a master - master relationship with replication


agreements between all servers.

I noticed the replication failure messages in the logs late last week 
and upon investigation found stale replication agreements for


ipa servers that had been replaced. Eventually I rebuilt 3 of the 4 
servers and re-initialized from the good server.


This morning my main ipa server had the directory service crash. After 
we restarted it, ipa-manage-replica --list-ruv showed


entries like these:

unable to decode: {replica 6} 55e49446 55e49446
unable to decode: {replica 4} 550b2d9e00020004 550b2d9e00020004
this happened when the ruv was recreated after a crash and the changelog 
contained references to cleaned RIDs. This is fixed in recent DS 
releases, the cleanallruv task now also cleans the changelog.


Which a cleanallruv.pl was able to remove.

We also noticed these log errors:

[07/Jun/2016:07:40:12 -0400] NSMMReplicationPlugin - ruv_compare_ruv: 
RUV [changelog max RUV] does not contain element
[{replica 1080 ldap://ipa1.p10jax.auth.monetra.com:389} 
57506ee60438 57506f0600160438] which is present in

RUV [database RUV]
[07/Jun/2016:07:40:12 -0400] NSMMReplicationPlugin - ruv_compare_ruv: 
RUV [changelog max RUV] does not contain element
[{replica 1285 ldap://ipa1.gnv.auth.monetra.com:389} 
5734e4730505 57361df00505] which is present in

RUV [database RUV]
[07/Jun/2016:07:40:12 -0400] NSMMReplicationPlugin - ruv_compare_ruv: 
RUV [changelog max RUV] does not contain element
[{replica 1085 ldap://ipa1.p10jax.auth.monetra.com:389} 
56d0aa27043d 57489fdd0003043d] which is present in

RUV [database RUV]
[07/Jun/2016:07:40:12 -0400] NSMMReplicationPlugin - ruv_compare_ruv: 
RUV [changelog max RUV] does not contain element


The cleanallruv script had no effect on these errors.
This are not really errors, it only indicates that the changelog does 
not (yet) contain chnages for specific RIDs), this could happen if the 
changelog was recreated, eg if after a crash it no longer matched the 
database. They should go away once the server has received changes for 
these RIDs


What is the proper procedure to clean up these stale entries? Is there 
something that I may be doing that causes this situation?


Thanks,

Andy



--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Can't set nsslapd-sizelimit

[Ludwig Krispenz]

On 05/17/2016 12:49 PM, Ludwig Krispenz wrote:


On 05/16/2016 11:19 PM, Giuseppe Sarno wrote:


Hello,

I am new to freeIPA and I am recently working on a project to 
integrate freeIPA with some legacy application which uses LDAP for 
user management.


I have initially created our own ldap structure and I tried to run 
the code against freeIPA/389DS. While running this example I noticed 
that 389DS takes quite some time to load profile data from the 
different ldap nodes (~2000 entries). In a previous prototype using 
OpenDJ we had to increase the parameter ds-cfg-size-limit: to ~1000 
with good results. I am wondering now whether we can do the same for 
the freeIPA/389DS server. I found the following pages but I could not 
work out what the exact command should be to modify those parameters.


https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/8.2/html/Administration_Guide/User_Account_Management-Setting_Resource_Limits_Based_on_the_Bind_DN.html

http://directory.fedoraproject.org/docs/389ds/howto/howto-ldapsearchmanyattr.html

I attempted the following but received a ObjectClass violation:

[centos@ldap-389ds-ireland ~]$ ldapmodify  -h ldap-389ds-ip -D 
"cn=Directory Manager" -w '' -f slimit


modifying entry "dc=ldap,dc=adeptra,dc=com"

ldap_modify: Object class violation (65)

additional info: attribute "nsslapd-sizelimit" not allowed

slimit:

dn: dc=ldap,dc=example,dc=com

changetype: modify

add:nsslapd-sizelimit

nsslapd-sizelimit: 1000

I also attempted using a user dn but with the same result.

the example in the doc is unfortunately incorrect, 
in the latest doc it is corected: 
https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/10/html/Administration_Guide/Finding_Directory_Entries.html#Setting_Resource_Limits_Based_on_the_Bind_DN-Setting_Resource_Limits_Using_the_Command_Line
nsslapd-sizelimit is the general limit in cn=config, the attribute per 
user is nsSizeLimit ( as used in the text in teh doc).

And you have to add it to a user used for binding


Can anybody help ?

Thanks,

Giuseppe.


Fair Isaac Services Limited (Co. No. 01998476) and Fair Isaac 
(Adeptra) Limited (Co. No. 03295455) are registered in England and 
Wales and have a registered office address of Cottons Centre, 5th 
Floor, Hays Lane, London, SE1 2QP.


This email and any files transmitted with it are confidential, 
proprietary and intended solely for the individual or entity to whom 
they are addressed. If you have received this email in error please 
delete it immediately.





--
Red Hat GmbH,http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael 
O'Neill




--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael 
O'Neill

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Can't set nsslapd-sizelimit

[Ludwig Krispenz]

On 05/16/2016 11:19 PM, Giuseppe Sarno wrote:


Hello,

I am new to freeIPA and I am recently working on a project to 
integrate freeIPA with some legacy application which uses LDAP for 
user management.


I have initially created our own ldap structure and I tried to run the 
code against freeIPA/389DS. While running this example I noticed that 
389DS takes quite some time to load profile data from the different 
ldap nodes (~2000 entries). In a previous prototype using OpenDJ we 
had to increase the parameter ds-cfg-size-limit: to ~1000 with good 
results. I am wondering now whether we can do the same for the 
freeIPA/389DS server. I found the following pages but I could not work 
out what the exact command should be to modify those parameters.


https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/8.2/html/Administration_Guide/User_Account_Management-Setting_Resource_Limits_Based_on_the_Bind_DN.html

http://directory.fedoraproject.org/docs/389ds/howto/howto-ldapsearchmanyattr.html

I attempted the following but received a ObjectClass violation:

[centos@ldap-389ds-ireland ~]$ ldapmodify  -h ldap-389ds-ip -D 
"cn=Directory Manager" -w '' -f slimit


modifying entry "dc=ldap,dc=adeptra,dc=com"

ldap_modify: Object class violation (65)

additional info: attribute "nsslapd-sizelimit" not allowed

slimit:

dn: dc=ldap,dc=example,dc=com

changetype: modify

add:nsslapd-sizelimit

nsslapd-sizelimit: 1000

I also attempted using a user dn but with the same result.

the example in the doc is unfortunately incorrect, nsslapd-sizelimit is 
the general limit in cn=config, the attribute per user is nsSizeLimit ( 
as used in the text in teh doc).

And you have to add it to a user used for binding


Can anybody help ?

Thanks,

Giuseppe.


Fair Isaac Services Limited (Co. No. 01998476) and Fair Isaac 
(Adeptra) Limited (Co. No. 03295455) are registered in England and 
Wales and have a registered office address of Cottons Centre, 5th 
Floor, Hays Lane, London, SE1 2QP.


This email and any files transmitted with it are confidential, 
proprietary and intended solely for the individual or entity to whom 
they are addressed. If you have received this email in error please 
delete it immediately.





--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael 
O'Neill

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] krb5kdc service not starting

[Ludwig Krispenz]

On 05/12/2016 05:28 AM, Prasun Gera wrote:

Hi everyone,
I had a pretty similar failure on my replica yesterday. The replica 
was not reachable, and I asked someone to have a look at the system. 
They presumably rebooted it. When it came back up, ipactl wouldn't 
start, and the symptoms were pretty similar to those described in this 
thread. I followed the solution of copying dse.ldif.startOK 
to dse.ldif, and that started everything.
This is very strange, it should not be possible to loose a dse.ldif, 
although you are now teh second person reporting this. I have seen 0 
length dse.ldif.tmp if a VM was powerd off while ds was active, but from 
DS  point of view it is not possible to complete loos the dse.ldif.
The dse.ldif stores the configuration information including replication 
agreements and and when ever this is updated the new state is written to 
disk. The procedure is like this:

-create a dse.ldif.tmp (this is the only time a 0 byte dse.ldif* file exists
-write the config to dse.ldif.tmp
-rename dse.ldif to dse.ldif.bak
-rename dse.ldif.tmp to dse.ldif

So, if the machine or the server crashes during this process there 
should be always a dse.ldif.tmp or dse.ldif.bak containing the current 
or latest information. If anyone has an idea how on a VM when powering 
it off can completely loose these files I would like to know.
However, I see some errors in dirsrv's logs. It is constantly printing 
lines like "DSRetroclPlugin - delete_changerecord: could not delete 
change record 418295". Is that normal ?
Unfortunately it can be. If after a crash the beginning of the retro cl 
is incorrectly calculated, changelog trimming might try to remov no 
longer existing records, it is annoying but harmless, so far we have not 
further investigated how to prevent this.
How do I confirm that the replica is back and fully functional ? Why 
did this happen in the first place ?


On Wed, Apr 27, 2016 at 1:41 PM, Gady Notrica <gnotr...@candeal.com 
<mailto:gnotr...@candeal.com>> wrote:


All good!!!

Gady

-Original Message-
From: Alexander Bokovoy [mailto:aboko...@redhat.com
<mailto:aboko...@redhat.com>]
Sent: April 27, 2016 1:19 PM
    To: Gady Notrica
Cc: Ludwig Krispenz; freeipa-users@redhat.com
<mailto:freeipa-users@redhat.com>
Subject: Re: [Freeipa-users] krb5kdc service not starting

On Wed, 27 Apr 2016, Gady Notrica wrote:
>Hello Ludwig,
>
>Is there a reason why my AD show offline?
>
>[root@cd-p-ipa1 /]# wbinfo --online-status BUILTIN : online IPA :
>online CD-PRD : offline
wbinfo output is irrelevant for RHEL 7.2-based IPA trusts.

You need to make sure that 'getent passwd CD-PRD\\Administrator'
resolves via SSSD.

--
/ Alexander Bokovoy

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project






--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael 
O'Neill

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA vulnerability management SSL

[Ludwig Krispenz]

wanted to add Noriko, but hit send to quickly

On 04/28/2016 01:26 PM, Ludwig Krispenz wrote:


On 04/28/2016 12:06 PM, Martin Kosek wrote:

On 04/28/2016 01:23 AM, Sean Hogan wrote:

Hi Martin,

No joy on placing - in front of the RC4s


I modified my nss.conf to now read
# SSL 3 ciphers. SSL 2 is disabled by default.
NSSCipherSuite
+aes_128_sha_256,+aes_256_sha_256,+ecdhe_ecdsa_aes_128_gcm_sha_256,+ecdhe_ecdsa_aes_128_sha,+ecdhe_ecdsa_aes_256_sha,+ecdhe_rsa_aes_128_gcm_sha_256,+ecdhe_rsa_aes_128_sha,+ecdhe_rsa_aes_256_sha,+rsa_aes_128_gcm_sha_256,+rsa_aes_128_sha,+rsa_aes_256_sha 



# SSL Protocol:
# Cryptographic protocols that provide communication security.
# NSS handles the specified protocols as "ranges", and automatically
# negotiates the use of the strongest protocol for a connection 
starting
# with the maximum specified protocol and downgrading as necessary 
to the

# minimum specified protocol that can be used between two processes.
# Since all protocol ranges are completely inclusive, and no 
protocol in the

NSSProtocol TLSv1.0,TLSv1.1,TLSv1.2

dse.ldif

dn: cn=encryption,cn=config
objectClass: top
objectClass: nsEncryptionConfig
cn: encryption
nsSSLSessionTimeout: 0
nsSSLClientAuth: allowed
nsSSL2: off
nsSSL3: off
creatorsName: cn=server,cn=plugins,cn=config
modifiersName: cn=directory manager
createTimestamp: 20150420131850Z
modifyTimestamp: 20150420131906Z
nsSSL3Ciphers: 
+all,-rsa_null_sha,-rsa_rc4_56_sha,-tls_rsa_export1024_with_rc4

_56_sha,-tls_dhe_dss_1024_rc4_sha
numSubordinates: 1



But I still get this with nmap.. I thought the above would remove
-tls_rsa_export1024_with_rc4_56_sha but still showing. Is it the 
fact that I am not
offering -tls_rsa_export1024_with_rc4_56_sha? If so.. not really 
understanding
where it is coming from cept the +all from DS but the - should be 
negating that?


Starting Nmap 5.51 ( http://nmap.org <http://nmap.org/> ) at 
2016-04-27 17:37 EDT

Nmap scan report for rtpvxl0077.watson.local (10.110.76.242)
Host is up (0.86s latency).
PORT STATE SERVICE
636/tcp open ldapssl
| ssl-enum-ciphers:
| TLSv1.2
| Ciphers (13)
| SSL_RSA_FIPS_WITH_3DES_EDE_CBC_SHA
| SSL_RSA_FIPS_WITH_DES_CBC_SHA
| TLS_RSA_EXPORT1024_WITH_DES_CBC_SHA
| TLS_RSA_EXPORT1024_WITH_RC4_56_SHA
| TLS_RSA_WITH_3DES_EDE_CBC_SHA
| TLS_RSA_WITH_AES_128_CBC_SHA
| TLS_RSA_WITH_AES_128_CBC_SHA256
| TLS_RSA_WITH_AES_128_GCM_SHA256
| TLS_RSA_WITH_AES_256_CBC_SHA
| TLS_RSA_WITH_AES_256_CBC_SHA256
| TLS_RSA_WITH_DES_CBC_SHA
| TLS_RSA_WITH_RC4_128_MD5
| TLS_RSA_WITH_RC4_128_SHA
| Compressors (1)
|_ uncompressed

Nmap done: 1 IP address (1 host up) scanned in 0.32 seconds



It seems no matter what config I put into nss.conf or dse.ldif 
nothing changes
with my nmap results. Is there supposed to be a be a section to add 
TLS ciphers

instead of SSL

Not sure now, CCing Ludwig who was involved in the original RHEL-6
implementation.
If I remember correctly we did the change in default ciphers and the 
option for handling in 389-ds > 1.3.3, so it would not be in RHEL6, 
adding Noriko to get confirmation.


but the below comments about changing ciphers in dse.ldif could help 
in using the "old" way to set ciphers

Just to be sure, when you are modifying dse.ldif, the procedure
should be always following:

1) Stop Directory Server service
2) Modify dse.ldif
3) Start Directory Server service

Otherwise it won't get applied and will get overwritten later.

In any case, the ciphers with RHEL-6 should be secure enough, the 
ones in
FreeIPA 4.3.1 should be even better. This is for example an nmap 
taken on

FreeIPA Demo instance that runs on FreeIPA 4.3.1:

$ nmap --script ssl-enum-ciphers -p 636 ipa.demo1.freeipa.org

Starting Nmap 7.12 ( https://nmap.org ) at 2016-04-28 12:02 CEST
Nmap scan report for ipa.demo1.freeipa.org (209.132.178.99)
Host is up (0.18s latency).
PORTSTATE SERVICE
636/tcp open  ldapssl
| ssl-enum-ciphers:
|   TLSv1.2:
| ciphers:
|   TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (secp256r1) - A
|   TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A
|   TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (secp256r1) - A
|   TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A
|   TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (dh 2048) - A
|   TLS_DHE_RSA_WITH_AES_128_CBC_SHA (dh 2048) - A
|   TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 (dh 2048) - A
|   TLS_DHE_RSA_WITH_AES_256_CBC_SHA (dh 2048) - A
|   TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 (dh 2048) - A
|   TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 2048) - A
|   TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
|   TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 2048) - A
|   TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A
|   TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 2048) - A
| compressors:
|   NULL
| cipher preference: server
|_  least strength: A

Nmap done: 1 IP address (1 host up) scanned in 21.12 seconds

Martin




--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Com

Re: [Freeipa-users] IPA vulnerability management SSL

[Ludwig Krispenz]

On 04/28/2016 12:06 PM, Martin Kosek wrote:

On 04/28/2016 01:23 AM, Sean Hogan wrote:

Hi Martin,

No joy on placing - in front of the RC4s


I modified my nss.conf to now read
# SSL 3 ciphers. SSL 2 is disabled by default.
NSSCipherSuite
+aes_128_sha_256,+aes_256_sha_256,+ecdhe_ecdsa_aes_128_gcm_sha_256,+ecdhe_ecdsa_aes_128_sha,+ecdhe_ecdsa_aes_256_sha,+ecdhe_rsa_aes_128_gcm_sha_256,+ecdhe_rsa_aes_128_sha,+ecdhe_rsa_aes_256_sha,+rsa_aes_128_gcm_sha_256,+rsa_aes_128_sha,+rsa_aes_256_sha

# SSL Protocol:
# Cryptographic protocols that provide communication security.
# NSS handles the specified protocols as "ranges", and automatically
# negotiates the use of the strongest protocol for a connection starting
# with the maximum specified protocol and downgrading as necessary to the
# minimum specified protocol that can be used between two processes.
# Since all protocol ranges are completely inclusive, and no protocol in the
NSSProtocol TLSv1.0,TLSv1.1,TLSv1.2

dse.ldif

dn: cn=encryption,cn=config
objectClass: top
objectClass: nsEncryptionConfig
cn: encryption
nsSSLSessionTimeout: 0
nsSSLClientAuth: allowed
nsSSL2: off
nsSSL3: off
creatorsName: cn=server,cn=plugins,cn=config
modifiersName: cn=directory manager
createTimestamp: 20150420131850Z
modifyTimestamp: 20150420131906Z
nsSSL3Ciphers: +all,-rsa_null_sha,-rsa_rc4_56_sha,-tls_rsa_export1024_with_rc4
_56_sha,-tls_dhe_dss_1024_rc4_sha
numSubordinates: 1



But I still get this with nmap.. I thought the above would remove
-tls_rsa_export1024_with_rc4_56_sha but still showing. Is it the fact that I am 
not
offering -tls_rsa_export1024_with_rc4_56_sha? If so.. not really understanding
where it is coming from cept the +all from DS but the - should be negating that?

Starting Nmap 5.51 ( http://nmap.org  ) at 2016-04-27 17:37 
EDT
Nmap scan report for rtpvxl0077.watson.local (10.110.76.242)
Host is up (0.86s latency).
PORT STATE SERVICE
636/tcp open ldapssl
| ssl-enum-ciphers:
| TLSv1.2
| Ciphers (13)
| SSL_RSA_FIPS_WITH_3DES_EDE_CBC_SHA
| SSL_RSA_FIPS_WITH_DES_CBC_SHA
| TLS_RSA_EXPORT1024_WITH_DES_CBC_SHA
| TLS_RSA_EXPORT1024_WITH_RC4_56_SHA
| TLS_RSA_WITH_3DES_EDE_CBC_SHA
| TLS_RSA_WITH_AES_128_CBC_SHA
| TLS_RSA_WITH_AES_128_CBC_SHA256
| TLS_RSA_WITH_AES_128_GCM_SHA256
| TLS_RSA_WITH_AES_256_CBC_SHA
| TLS_RSA_WITH_AES_256_CBC_SHA256
| TLS_RSA_WITH_DES_CBC_SHA
| TLS_RSA_WITH_RC4_128_MD5
| TLS_RSA_WITH_RC4_128_SHA
| Compressors (1)
|_ uncompressed

Nmap done: 1 IP address (1 host up) scanned in 0.32 seconds



It seems no matter what config I put into nss.conf or dse.ldif nothing changes
with my nmap results. Is there supposed to be a be a section to add TLS ciphers
instead of SSL

Not sure now, CCing Ludwig who was involved in the original RHEL-6
implementation.
If I remember correctly we did the change in default ciphers and the 
option for handling in 389-ds > 1.3.3, so it would not be in RHEL6, 
adding Noriko to get confirmation.


but the below comments about changing ciphers in dse.ldif could help in 
using the "old" way to set ciphers

Just to be sure, when you are modifying dse.ldif, the procedure
should be always following:

1) Stop Directory Server service
2) Modify dse.ldif
3) Start Directory Server service

Otherwise it won't get applied and will get overwritten later.

In any case, the ciphers with RHEL-6 should be secure enough, the ones in
FreeIPA 4.3.1 should be even better. This is for example an nmap taken on
FreeIPA Demo instance that runs on FreeIPA 4.3.1:

$ nmap --script ssl-enum-ciphers -p 636 ipa.demo1.freeipa.org

Starting Nmap 7.12 ( https://nmap.org ) at 2016-04-28 12:02 CEST
Nmap scan report for ipa.demo1.freeipa.org (209.132.178.99)
Host is up (0.18s latency).
PORTSTATE SERVICE
636/tcp open  ldapssl
| ssl-enum-ciphers:
|   TLSv1.2:
| ciphers:
|   TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (secp256r1) - A
|   TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A
|   TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (secp256r1) - A
|   TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A
|   TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (dh 2048) - A
|   TLS_DHE_RSA_WITH_AES_128_CBC_SHA (dh 2048) - A
|   TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 (dh 2048) - A
|   TLS_DHE_RSA_WITH_AES_256_CBC_SHA (dh 2048) - A
|   TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 (dh 2048) - A
|   TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 2048) - A
|   TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
|   TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 2048) - A
|   TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A
|   TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 2048) - A
| compressors:
|   NULL
| cipher preference: server
|_  least strength: A

Nmap done: 1 IP address (1 host up) scanned in 21.12 seconds

Martin


--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael 
O'Neill

--
Manage your 

Re: [Freeipa-users] krb5kdc service not starting

[Ludwig Krispenz]

On 04/27/2016 05:10 PM, Gady Notrica wrote:


Oh! No…

Is there a way I can pull those files from the secondary server and 
put them on the primary?


do you have any file /etc/dirsrv/slapd-IPA-CANDEAL-CA/dse* ? There might 
be some older states to try
If you want to use a dse.ldif from another server, it could only work if 
the other server is really the same, same backends, indexes,, and 
you would have to do a lot of editing to adapt the file to the local 
system, eg replication agreements 

And then it is not sure if something else could be broken


Or I can run the re-installation ipa-server-install with repair option 
and copy the data back from the secondary server?


I'm not so sure about the IPA reinstall/repair process, maybe soemone 
else can step in


Thanks,

Gady Notrica| IT Systems Analyst | 416.814.7800 Ext. 7921 | Cell. 
416.818.4797 | gnotr...@candeal.com <mailto:gnotr...@candeal.com>


CanDeal | 152 King St. E, 4th Floor, Toronto ON M5A 1J4 | 
www.candeal.com <http://www.candeal.ca/>| Follow us:Description: 
Description: cid:image003.jpg@01CBD419.622CDF90 
<http://www.twitter.com/candeal>*Description: Description: 
Description: cid:image002.jpg@01CBD419.622CDF90* 
<http://www.linkedin.com/profile/view?id=36869324=tab_pro>


*From:*Ludwig Krispenz [mailto:lkris...@redhat.com]
*Sent:* April 27, 2016 10:58 AM
*To:* Gady Notrica
*Cc:* Rob Crittenden; freeipa-users@redhat.com
*Subject:* Re: [Freeipa-users] krb5kdc service not starting

On 04/27/2016 04:36 PM, Gady Notrica wrote:

*No changes*to /var/log/dirsrv/slapd-IPA-CANDEAL-CA/errors. I am
tailing the log file and running those commands doesn’t generate
any log, nothing.

[root@cd-p-ipa1 log]# ipactl start

Starting Directory Service

Job for dirsrv@IPA-CANDEAL-CA.service
<mailto:dirsrv@IPA-CANDEAL-CA.service> failed because the control
process exited with error code. See "systemctl status
dirsrv@IPA-CANDEAL-CA.service
<mailto:dirsrv@IPA-CANDEAL-CA.service>" and "journalctl -xe" for
details.

Failed to start Directory Service: Command ''/bin/systemctl'
'start' 'dirsrv@IPA-CANDEAL-CA.service
<mailto:dirsrv@IPA-CANDEAL-CA.service>'' returned non-zero exit
status 1

*Logs from /var/log/messages*

Apr 27 10:26:05 cd-p-ipa1 systemd: Starting 389 Directory Server
IPA-CANDEAL-CA

Apr 27 10:26:05 cd-p-ipa1 ns-slapd: [27/Apr/2016:10:26:05 -0400]
dse - The configuration file
/etc/dirsrv/slapd-IPA-CANDEAL-CA/dse.ldif was not restored from
backup /etc/dirsrv/slapd-IPA-CANDEAL-CA/dse.ldif.tmp, error -1

Apr 27 10:26:05 cd-p-ipa1 ns-slapd: [27/Apr/2016:10:26:05 -0400]
dse - The configuration file
/etc/dirsrv/slapd-IPA-CANDEAL-CA/dse.ldif was not restored from
backup /etc/dirsrv/slapd-IPA-CANDEAL-CA/dse.ldif.bak, error -1

Apr 27 10:26:05 cd-p-ipa1 ns-slapd: [27/Apr/2016:10:26:05 -0400]
config - The given config file
/etc/dirsrv/slapd-IPA-CANDEAL-CA/dse.ldif could not be accessed,
Netscape Portable Runtime error -5950 (File not found.)

this is BAD, looks like you completely lost your configuration file 
for DS, so it doesn't even know where to log anything. When you lost 
your VM and rebooted there must hav ebeen some data loss.

It could be only dse.ldif, but also other files.

[root@cd-p-ipa1 log]# systemctl start dirsrv@IPA-CANDEAL-CA.service 
<mailto:dirsrv@IPA-CANDEAL-CA.service>


Job for dirsrv@IPA-CANDEAL-CA.service 
<mailto:dirsrv@IPA-CANDEAL-CA.service> failed because the control 
process exited with error code. See "systemctl status 
dirsrv@IPA-CANDEAL-CA.service <mailto:dirsrv@IPA-CANDEAL-CA.service>" 
and "journalctl -xe" for details.


[root@cd-p-ipa1 log]# systemctl status dirsrv@IPA-CANDEAL-CA.service 
<mailto:dirsrv@IPA-CANDEAL-CA.service> -l


● dirsrv@IPA-CANDEAL-CA.service <mailto:dirsrv@IPA-CANDEAL-CA.service> 
- 389 Directory Server IPA-CANDEAL-CA.


Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service 
<mailto:/usr/lib/systemd/system/dirsrv@.service>; enabled; vendor 
preset: disabled)


Active: failed (Result: exit-code) since Wed 2016-04-27 10:26:17 EDT; 
3s ago


Process: 9830 ExecStart=/usr/sbin/ns-slapd -D /etc/dirsrv/slapd-%i -i 
/var/run/dirsrv/slapd-%i.pid -w /var/run/dirsrv/slapd-%i.startpid 
(code=exited, status=1/FAILURE)


Apr 27 10:26:17 cd-p-ipa1.ipa.candeal.ca ns-slapd[9830]: 
[27/Apr/2016:10:26:17 -0400] - valueset_value_syntax_cmp: 
slapi_attr_values2keys_sv failed for type attributetypes


Apr 27 10:26:17 cd-p-ipa1.ipa.candeal.ca ns-slapd[9830]: 
[27/Apr/2016:10:26:17 -0400] - valueset_value_syntax_cmp: 
slapi_attr_values2keys_sv failed for type attributetypes


Apr 27 10:26:17 cd-p-ipa1.ipa.candeal.ca ns-slapd[9830]: 
[27/Apr/2016:10:26:17 -0400] - valueset_value_syntax_cmp: 
slapi_attr_values2keys_sv failed for type attributetypes


Apr 27 10:26:17 cd-p-ipa

Re: [Freeipa-users] krb5kdc service not starting

[Ludwig Krispenz]

On 04/27/2016 04:36 PM, Gady Notrica wrote:


*No changes*to /var/log/dirsrv/slapd-IPA-CANDEAL-CA/errors. I am 
tailing the log file and running those commands doesn’t generate any 
log, nothing.


[root@cd-p-ipa1 log]# ipactl start

Starting Directory Service

Job for dirsrv@IPA-CANDEAL-CA.service failed because the control 
process exited with error code. See "systemctl status 
dirsrv@IPA-CANDEAL-CA.service" and "journalctl -xe" for details.


Failed to start Directory Service: Command ''/bin/systemctl' 'start' 
'dirsrv@IPA-CANDEAL-CA.service'' returned non-zero exit status 1


*Logs from /var/log/messages*

Apr 27 10:26:05 cd-p-ipa1 systemd: Starting 389 Directory Server 
IPA-CANDEAL-CA


Apr 27 10:26:05 cd-p-ipa1 ns-slapd: [27/Apr/2016:10:26:05 -0400] dse - 
The configuration file /etc/dirsrv/slapd-IPA-CANDEAL-CA/dse.ldif was 
not restored from backup 
/etc/dirsrv/slapd-IPA-CANDEAL-CA/dse.ldif.tmp, error -1


Apr 27 10:26:05 cd-p-ipa1 ns-slapd: [27/Apr/2016:10:26:05 -0400] dse - 
The configuration file /etc/dirsrv/slapd-IPA-CANDEAL-CA/dse.ldif was 
not restored from backup 
/etc/dirsrv/slapd-IPA-CANDEAL-CA/dse.ldif.bak, error -1


Apr 27 10:26:05 cd-p-ipa1 ns-slapd: [27/Apr/2016:10:26:05 -0400] 
config - The given config file 
/etc/dirsrv/slapd-IPA-CANDEAL-CA/dse.ldif could not be accessed, 
Netscape Portable Runtime error -5950 (File not found.)


this is BAD, looks like you completely lost your configuration file for 
DS, so it doesn't even know where to log anything. When you lost your VM 
and rebooted there must hav ebeen some data loss.

It could be only dse.ldif, but also other files.


[root@cd-p-ipa1 log]# systemctl start dirsrv@IPA-CANDEAL-CA.service

Job for dirsrv@IPA-CANDEAL-CA.service failed because the control 
process exited with error code. See "systemctl status 
dirsrv@IPA-CANDEAL-CA.service" and "journalctl -xe" for details.


[root@cd-p-ipa1 log]# systemctl status dirsrv@IPA-CANDEAL-CA.service -l

● dirsrv@IPA-CANDEAL-CA.service - 389 Directory Server IPA-CANDEAL-CA.

Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; 
vendor preset: disabled)


Active: failed (Result: exit-code) since Wed 2016-04-27 10:26:17 EDT; 
3s ago


Process: 9830 ExecStart=/usr/sbin/ns-slapd -D /etc/dirsrv/slapd-%i -i 
/var/run/dirsrv/slapd-%i.pid -w /var/run/dirsrv/slapd-%i.startpid 
(code=exited, status=1/FAILURE)


Apr 27 10:26:17 cd-p-ipa1.ipa.candeal.ca ns-slapd[9830]: 
[27/Apr/2016:10:26:17 -0400] - valueset_value_syntax_cmp: 
slapi_attr_values2keys_sv failed for type attributetypes


Apr 27 10:26:17 cd-p-ipa1.ipa.candeal.ca ns-slapd[9830]: 
[27/Apr/2016:10:26:17 -0400] - valueset_value_syntax_cmp: 
slapi_attr_values2keys_sv failed for type attributetypes


Apr 27 10:26:17 cd-p-ipa1.ipa.candeal.ca ns-slapd[9830]: 
[27/Apr/2016:10:26:17 -0400] - valueset_value_syntax_cmp: 
slapi_attr_values2keys_sv failed for type attributetypes


Apr 27 10:26:17 cd-p-ipa1.ipa.candeal.ca ns-slapd[9830]: 
[27/Apr/2016:10:26:17 -0400] - valueset_value_syntax_cmp: 
slapi_attr_values2keys_sv failed for type attributetypes


Apr 27 10:26:17 cd-p-ipa1.ipa.candeal.ca ns-slapd[9830]: 
[27/Apr/2016:10:26:17 -0400] - valueset_value_syntax_cmp: 
slapi_attr_values2keys_sv failed for type attributetypes


Apr 27 10:26:17 cd-p-ipa1.ipa.candeal.ca ns-slapd[9830]: 
[27/Apr/2016:10:26:17 -0400] - valueset_value_syntax_cmp: 
slapi_attr_values2keys_sv failed for type attributetypes


Apr 27 10:26:17 cd-p-ipa1.ipa.candeal.ca ns-slapd[9830]: 
[27/Apr/2016:10:26:17 -0400] - valueset_value_syntax_cmp: 
slapi_attr_values2keys_sv failed for type attributetypes


Apr 27 10:26:17 cd-p-ipa1.ipa.candeal.ca ns-slapd[9830]: 
[27/Apr/2016:10:26:17 -0400] - valueset_value_syntax_cmp: 
slapi_attr_values2keys_sv failed for type attributetypes


Apr 27 10:26:17 cd-p-ipa1.ipa.candeal.ca ns-slapd[9830]: 
[27/Apr/2016:10:26:17 -0400] dse_read_one_file - The entry cn=schema 
in file /etc/dirsrv/slapd-IPA-CANDEAL-CA/schema/00core.ldif (lineno: 
1) is invalid, error code 21 (Invalid syntax) - attribute type aci: 
Unknown attribute syntax OID "1.3.6.1.4.1.1466.115.121.1.15"


Apr 27 10:26:17 cd-p-ipa1.ipa.candeal.ca ns-slapd[9830]: 
[27/Apr/2016:10:26:17 -0400] dse - Please edit the file to correct the 
reported problems and then restart the server.


[root@cd-p-ipa1 log]#

Gady

*From:*Ludwig Krispenz [mailto:lkris...@redhat.com]
*Sent:* April 27, 2016 10:06 AM
*To:* Gady Notrica
*Cc:* Rob Crittenden; freeipa-users@redhat.com
*Subject:* Re: [Freeipa-users] krb5kdc service not starting

On 04/27/2016 03:48 PM, Gady Notrica wrote:

Hello Ludwig,

I do have only 1 error logs for the 26^th in
/var/log/dirsrv/slapd-IPA-CANDEAL-CA/errors. Below is the only
line I have

[25/Apr/2016:22:34:51 -0400] NSMMReplicationPlugin - windows sync
- failed to send dirsync search request: 2

[*26/Apr/2016*:00:13:01 -0400] - Entry
"uid=MMOOREDT$,cn=users,cn=accounts,dc=ip

Re: [Freeipa-users] krb5kdc service not starting

[Ludwig Krispenz]

On 04/26/2016 09:09 PM, Gady Notrica wrote:


HERE..

[23/Apr/2016:11:39:51 -0400] set_krb5_creds - Could not get initial 
credentials for principal 
[ldap/cd-p-ipa1.ipa.domain.local@IPA.DOMAIN.LOCAL] in keytab 
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for 
requested realm)


[23/Apr/2016:11:39:51 -0400] slapd_ldap_sasl_interactive_bind - Error: 
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 
-2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified 
GSS failure.  Minor code may provide more information (No Kerberos 
credentials available)) errno 0 (Success)


[23/Apr/2016:11:39:51 -0400] slapi_ldap_bind - Error: could not 
perform interactive bind for id [] authentication mechanism [GSSAPI]: 
error -2 (Local error)


[23/Apr/2016:11:39:51 -0400] NSMMReplicationPlugin - 
agmt="cn=meTocd-s-ipa1.ipa.domain.local" (cd-s-ipa1:389): Replication 
bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): 
generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code 
may provide more information (No Kerberos credentials available))


[23/Apr/2016:11:39:51 -0400] - slapd started.  Listening on All 
Interfaces port 389 for LDAP requests


[23/Apr/2016:11:39:51 -0400] - Listening on All Interfaces port 636 
for LDAPS requests


[23/Apr/2016:11:39:51 -0400] - Listening on 
/var/run/slapd-IPA-DOMAIN-LOCAL.socket for LDAPI requests


[23/Apr/2016:11:39:55 -0400] NSMMReplicationPlugin - 
agmt="cn=meTocd-s-ipa1.ipa.domain.local" (cd-s-ipa1:389): Replication 
bind with GSSAPI auth resumed


[23/Apr/2016:14:37:27 -0400] NSMMReplicationPlugin - 
agmt="cn=meTocd-s-ipa1.ipa.domain.local" (cd-s-ipa1:389): Unable to 
receive the response for a startReplication extended operation to 
consumer (Can't contact LDAP server). Will retry later.


[23/Apr/2016:14:38:02 -0400] slapd_ldap_sasl_interactive_bind - Error: 
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 
-1 (Can't contact LDAP server) ((null)) errno 107 (Transport endpoint 
is not connected)


[23/Apr/2016:14:38:02 -0400] slapi_ldap_bind - Error: could not 
perform interactive bind for id [] authentication mechanism [GSSAPI]: 
error -1 (Can't contact LDAP server)


[23/Apr/2016:14:38:02 -0400] slapd_ldap_sasl_interactive_bind - Error: 
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 
-1 (Can't contact LDAP server) ((null)) errno 107 (Transport endpoint 
is not connected)


[23/Apr/2016:14:38:02 -0400] slapi_ldap_bind - Error: could not 
perform interactive bind for id [] authentication mechanism [GSSAPI]: 
error -1 (Can't contact LDAP server)


[23/Apr/2016:14:38:02 -0400] slapd_ldap_sasl_interactive_bind - Error: 
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 
-1 (Can't contact LDAP server) ((null)) errno 107 (Transport endpoint 
is not connected)


[23/Apr/2016:14:38:02 -0400] slapi_ldap_bind - Error: could not 
perform interactive bind for id [] authentication mechanism [GSSAPI]: 
error -1 (Can't contact LDAP server)


[23/Apr/2016:14:38:13 -0400] NSMMReplicationPlugin - 
agmt="cn=meTocd-s-ipa1.ipa.domain.local" (cd-s-ipa1:389): Replication 
bind with GSSAPI auth resumed


[25/Apr/2016:22:34:51 -0400] NSMMReplicationPlugin - windows sync - 
failed to send dirsync search request: 2



these are old logs, the problem you were reporting was on Apr, 26:

Apr 26 08:50:21 cd-p-ipa1.ipa.candeal.ca ns-slapd[6333]: [26/Apr/2016:08:50:21 -0400] 
dse_read_one_file - The entry cn=schema in file 
/etc/dirsrv/slapd-IPA-CANDEAL-CA/schema/00core.ldif (lineno: 1) is invalid, error code 21 
(Invalid syntax) - attribute type aci: Unknown attribute syntax OID 
"1.3.6.1.4.1.1466.115.121.1.15"
Apr 26 08:50:21 cd-p-ipa1.ipa.candeal.ca ns-slapd[6333]: [26/Apr/2016:08:50:21 
-0400] dse - Please edit the file to correct the reported problems and then 
restart the server.


we need the logs from that time




Gady

-Original Message-
From: Rob Crittenden [mailto:rcrit...@redhat.com]
Sent: April 26, 2016 2:44 PM
To: Gady Notrica; Ludwig Krispenz; freeipa-users@redhat.com
Subject: Re: [Freeipa-users] krb5kdc service not starting

Gady Notrica wrote:

> Hey world,

>

> Any ideas?

What about the first part of Ludwig's question: Is there anything in 
the 389-ds error log?


rob

>

> Gady

>

> -Original Message-

> From: freeipa-users-boun...@redhat.com 
<mailto:freeipa-users-boun...@redhat.com>


> [mailto:freeipa-users-boun...@redhat.com] On Behalf Of Gady Notrica

> Sent: April 26, 2016 10:10 AM

> To: Ludwig Krispenz; freeipa-users@redhat.com 
<mailto:freeipa-users@redhat.com>


> Subject: Re: [Freeipa-users] krb5kdc service not starting

>

> No, no changes. Lost connectivity with my VMs during the night

> (networking issues in datacenter)

>

> Reboot the server and oups, no IPA is coming up... The replica 
(s

Re: [Freeipa-users] krb5kdc service not starting

[Ludwig Krispenz]

On 04/26/2016 03:26 PM, Gady Notrica wrote:

Here...

[root@cd-p-ipa1 log]# ipactl status
Directory Service: STOPPED
Directory Service must be running in order to obtain status of other services
ipa: INFO: The ipactl command was successful

[root@cd-p-ipa1 log]# systemctl status dirsrv@IPA-CANDEAL-CA.service -l
● dirsrv@IPA-CANDEAL-CA.service - 389 Directory Server IPA-CANDEAL-CA.
Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor 
preset: disabled)
Active: failed (Result: exit-code) since Tue 2016-04-26 08:50:21 EDT; 30min 
ago
   Process: 6333 ExecStart=/usr/sbin/ns-slapd -D /etc/dirsrv/slapd-%i -i 
/var/run/dirsrv/slapd-%i.pid -w /var/run/dirsrv/slapd-%i.startpid (code=exited, 
status=1/FAILURE)

Apr 26 08:50:21 cd-p-ipa1.ipa.candeal.ca ns-slapd[6333]: [26/Apr/2016:08:50:21 
-0400] - valueset_value_syntax_cmp: slapi_attr_values2keys_sv failed for type 
attributetypes
Apr 26 08:50:21 cd-p-ipa1.ipa.candeal.ca ns-slapd[6333]: [26/Apr/2016:08:50:21 
-0400] - valueset_value_syntax_cmp: slapi_attr_values2keys_sv failed for type 
attributetypes
Apr 26 08:50:21 cd-p-ipa1.ipa.candeal.ca ns-slapd[6333]: [26/Apr/2016:08:50:21 
-0400] - valueset_value_syntax_cmp: slapi_attr_values2keys_sv failed for type 
attributetypes
Apr 26 08:50:21 cd-p-ipa1.ipa.candeal.ca ns-slapd[6333]: [26/Apr/2016:08:50:21 
-0400] - valueset_value_syntax_cmp: slapi_attr_values2keys_sv failed for type 
attributetypes
Apr 26 08:50:21 cd-p-ipa1.ipa.candeal.ca ns-slapd[6333]: [26/Apr/2016:08:50:21 
-0400] - valueset_value_syntax_cmp: slapi_attr_values2keys_sv failed for type 
attributetypes
Apr 26 08:50:21 cd-p-ipa1.ipa.candeal.ca ns-slapd[6333]: [26/Apr/2016:08:50:21 
-0400] - valueset_value_syntax_cmp: slapi_attr_values2keys_sv failed for type 
attributetypes
Apr 26 08:50:21 cd-p-ipa1.ipa.candeal.ca ns-slapd[6333]: [26/Apr/2016:08:50:21 
-0400] - valueset_value_syntax_cmp: slapi_attr_values2keys_sv failed for type 
attributetypes
Apr 26 08:50:21 cd-p-ipa1.ipa.candeal.ca ns-slapd[6333]: [26/Apr/2016:08:50:21 
-0400] - valueset_value_syntax_cmp: slapi_attr_values2keys_sv failed for type 
attributetypes
Apr 26 08:50:21 cd-p-ipa1.ipa.candeal.ca ns-slapd[6333]: [26/Apr/2016:08:50:21 -0400] 
dse_read_one_file - The entry cn=schema in file 
/etc/dirsrv/slapd-IPA-CANDEAL-CA/schema/00core.ldif (lineno: 1) is invalid, error code 21 
(Invalid syntax) - attribute type aci: Unknown attribute syntax OID 
"1.3.6.1.4.1.1466.115.121.1.15"
Apr 26 08:50:21 cd-p-ipa1.ipa.candeal.ca ns-slapd[6333]: [26/Apr/2016:08:50:21 
-0400] dse - Please edit the file to correct the reported problems and then 
restart the server.
this says the server doesn't know a syntax oid, but it is a known one. 
It could be that the syntax plugings couldn't be loaded. Thera are more 
errors before, could you check where the errors start in 
/var/log/dirsrv/slapd-/errors ?


And, did you do any changes to the system before this problem started ?

[root@cd-p-ipa1 log]#

Gady

-Original Message-
From: freeipa-users-boun...@redhat.com 
[mailto:freeipa-users-boun...@redhat.com] On Behalf Of Martin Babinsky
Sent: April 26, 2016 9:17 AM
To: freeipa-users@redhat.com
Subject: Re: [Freeipa-users] krb5kdc service not starting

On 04/26/2016 03:13 PM, Gady Notrica wrote:

Hello world,



I am having issues this morning with my primary IPA. See below the
details in the logs and command result. Basically, krb5kdc service not
starting - krb5kdc: Server error - while fetching master key.



DNS is functioning. See below dig result. I have a trust with Windows AD.



Please help…!



[root@cd-ipa1 log]# systemctl status krb5kdc.service -l

● krb5kdc.service - Kerberos 5 KDC

Loaded: loaded (/usr/lib/systemd/system/krb5kdc.service; disabled;
vendor preset: disabled)

Active: failed (Result: exit-code) since Tue 2016-04-26 08:27:52
EDT; 41min ago

   Process: 3694 ExecStart=/usr/sbin/krb5kdc -P /var/run/krb5kdc.pid
$KRB5KDC_ARGS (code=exited, status=1/FAILURE)



Apr 26 08:27:52 cd-ipa1.ipa.domain.localsystemd[1]: Starting Kerberos
5 KDC...

Apr 26 08:27:52 cd-ipa1.ipa.domain.localkrb5kdc[3694]: krb5kdc: cannot
initialize realm IPA.DOMAIN.LOCAL- see log file for details

Apr 26 08:27:52 cd-ipa1.ipa.domain.localsystemd[1]: krb5kdc.service:
control process exited, code=exited status=1

Apr 26 08:27:52 cd-ipa1.ipa.domain.localsystemd[1]: Failed to start
Kerberos 5 KDC.

Apr 26 08:27:52 cd-ipa1.ipa.domain.localsystemd[1]: Unit
krb5kdc.service entered failed state.

Apr 26 08:27:52 cd-ipa1.ipa.domain.localsystemd[1]: krb5kdc.service failed.

[root@cd-ipa1 log]#



Errors in /var/log/krb5kdc.log



krb5kdc: Server error - while fetching master key K/M for realm
DOMAIN.LOCAL

krb5kdc: Server error - while fetching master key K/M for realm
DOMAIN.LOCAL

krb5kdc: Server error - while fetching master key K/M for realm
DOMAIN.LOCAL



[root@cd-ipa1 log]# systemctl status httpd -l

● httpd.service - The Apache HTTP Server

Loaded: loaded 

Re: [Freeipa-users] Error setting up Replication: ldap service principals is missing. Replication agreement cannot be converted

[Ludwig Krispenz]

On 04/15/2016 10:14 AM, Kilian Ries wrote:

Hi,

on auht01 i see the following error just before installation fails:


[14/Apr/2016:15:57:09 +0200] - database index operation failed BAD 1031, 
err= Unknown error 
[14/Apr/2016:15:57:09 +0200] - add: attempt to index 625 failed; rc=
[14/Apr/2016:15:57:09 +0200] - str2entry_fast: entry has no dn
[14/Apr/2016:15:57:09 +0200] id2entry - str2entry returned NULL for id 252, 
string=""
[14/Apr/2016:15:57:09 +0200] - dn2entry_ext: the dn 
"krbprincipalname=ldap/auth02.intern...@intern.eu,cn=services,cn=accounts,dc=intern,dc=eu"
 was in the entryrdn index, but it did not exist in id2entry of instance userRoot.
[14/Apr/2016:15:57:09 +0200] entryrdn-index - _entryrdn_insert_key: Same DN 
(dn: 
krbprincipalname=ldap/auth02.intern...@intern.eu,cn=services,cn=accounts,dc=intern,dc=eu)
 is already in the entryrdn file with different ID 252.  Expected ID is 625.
[14/Apr/2016:15:57:09 +0200] - database index operation failed BAD 1031, 
err= Unknown error 
[14/Apr/2016:15:57:09 +0200] - add: attempt to index 625 failed; rc=
[14/Apr/2016:15:57:19 +0200] - str2entry_fast: entry has no dn
[14/Apr/2016:15:57:19 +0200] id2entry - str2entry returned NULL for id 252, 
string=""
[14/Apr/2016:15:57:21 +0200] - str2entry_fast: entry has no dn
this looks like a database/index corruption. There are traces for the 
ldapprincipal for auth02in the database, but teh index and the database 
are inconsistent. you can try to reindex teh database and see if this helps:

db2index.pl -D ... -w .. -Z  -t entryrdn  #only this index
or
db2index.pl -D ... -w .. -Z  # full reindex



[14/Apr/2016:16:02:01 +0200] attrlist_replace - attr_replace (nsslapd-referral, 
ldap://auth02.intern.eu:389/o%3Dipaca) failed.


Greets
Kilian



Von: freeipa-users-boun...@redhat.com <freeipa-users-boun...@redhat.com> im Auftrag 
von Ludwig Krispenz <lkris...@redhat.com>
Gesendet: Donnerstag, 14. April 2016 16:46
An: freeipa-users@redhat.com
Betreff: Re: [Freeipa-users] Error setting up Replication: ldap service 
principals is missing. Replication agreement cannot be converted

On 04/14/2016 04:19 PM, Kilian Ries wrote:

Hello Rob,

thanks for your explanations. I followed your hints and did a complete 
uninstall and started over with a fresh installation. I ended up with exactly 
the same error as the first time...

I did the following steps:


auth01$ ipa-replica-manage del auth02

auth02$ ipa-server-install --uninstall

auth01$ ipa-replica-prepare --ip-address 192.168.210.181 auth02.intern.eu

auth02$ ipa-replica-install --setup-dns --setup-ca --forwarder 192.168.210.40 
/root/replica-info-auth02.intern.eu.gpg


Are there other logfiles i can check for more specific errors?

you should have a look to the DS error logs in /var/log/dirsrv on both
instances

Greets
Kilian


Von: Rob Crittenden <rcrit...@redhat.com>
Gesendet: Mittwoch, 13. April 2016 16:18
An: Kilian Ries; freeipa-users@redhat.com
Betreff: Re: [Freeipa-users] Error setting up Replication: ldap service 
principals is missing. Replication agreement cannot be converted

Kilian Ries wrote:

Does nobody have an idea whats the problem here?

TL;DR you are best off deleting this failed replica install and trying
again.

Initial replication is done over TLS. When replication is completed both
sides of the agreement are converted to using GSSAPI and both ldap
principals are needed to do this. Given that replication just completed
both principals should be available but rarely one is not (hence the
vague-ish error message).

In this case the new ldap principal for the new replica wasn't found on
the remote master so things blew up.

There is no continuing the installation after this type of failure so
you'll need to remove the failed install as a master on auth01
(ipa-replica-manage del auth02...) and then run ipa-server-install
--uninstall on autho02 and try again.

rob


Thanks

Kilian




*Von:* freeipa-users-boun...@redhat.com
<freeipa-users-boun...@redhat.com> im Auftrag von Kilian Ries
<m...@kilian-ries.de>
*Gesendet:* Mittwoch, 6. April 2016 10:41
*An:* freeipa-users@redhat.com
*Betreff:* [Freeipa-users] Error setting up Replication: ldap service
principals is missing. Replication agreement cannot be converted

Hello,


i have an existing FreeIPA installation (4.2.0) on CentOS 7.2 and i'm
trying to add an replication partner.


During the installation i got the following error:


###

Restarting the directory and certificate servers

Configuring Kerberos KDC (krb5kdc). Estimated time: 30 seconds

 [1/8]: adding sasl mappings to the directory

 [2/8]: configuring KDC

 [3/8]: creating a keytab for the directory

 [4/8]: creating a keytab for the machine

 [5/8]: adding the password extension to the direc

Re: [Freeipa-users] Zombie Replica !

[Ludwig Krispenz]

On 04/07/2016 07:23 AM, Prashant Bapat wrote:
What I have done now was to add a new server, ipa02 and configured 
replication again and things are fine.


However on IPA1 the 389 ds error logs have reference to the dead ipa2 
replica.


[07/Apr/2016:04:13:11 +] NSMMReplicationPlugin - 
agmt="cn=meToipa2.example.net " 
(ipa2:389): Replication bind with GSSAPI auth failed: LDAP error -1 
(Can't contact LDAP server) ()
[07/Apr/2016:04:13:11 +] NSMMReplicationPlugin - Abort CleanAllRUV 
Task (rid 6): Failed to connect to 
replica(agmt="cn=meToipa2.example.net " 
(ipa2:389)).
[07/Apr/2016:04:13:11 +] NSMMReplicationPlugin - Abort CleanAllRUV 
Task (rid 6): Retrying in 14400 seconds


It will never be able to connect to ipa2 as its gone permanently. 
Also the ipa-replica-manage list `hostname`command still shows the 
ipa2 as replica.


How to remove this permanently ???
I don't know why you did get into this state, ipa-replica-manage del 
should have removed the agreement. You can do it by directly deleting it 
in DS:

- get the full dn of the agreement
ldapsearch . -D "cn=directory manager" -w  -b cn=config 
"cn=meToipa2.example.net" dn 

it should return an entry with
dn: 

the do a delete

ldapmodify . -D "cn=directory manager" -w 
dn: 
changetype: delete



Thanks.
--Prashant

On 6 April 2016 at 22:17, Prashant Bapat > wrote:


# ipa-replica-manage list `hostname`
ipa2.example.net : replica
ipa3.example.net : replica
ipa4.example.net : replica

ipa2.example.net  should not be there.
How do I remove it?

On 6 April 2016 at 18:55, Rob Crittenden > wrote:

Prashant Bapat wrote:

Hi,

We had 4 IPA servers in master master mode with all of
them connected to
each other.

IPA1 <>  IPA2 (colo 1)
IPA3 <>  IPA4 (colo 2)

One of the replica servers (IPA2) had to be rebuild.

So I went ahead and used below commands.

ipa-replica-manage disconnect IPA2 IPA3
ipa-replica-manage disconnection IPA2 IPA4
ipa-replica-manage del IPA2 (to remove it on IPA1).

An then ran ipa-server-install --uninstallon IPA2.

Created the replica info file using ipa-replica-prepare IPA2.

When I tried to run ipa-replica-install on IPA2, it says

A replication agreement for this host already exists. It
needs to be
removed.
Run this on the master that generated the info file:
 % ipa-replica-manage del ipa2.example.net
 
--force

Now on IPA1, no matter what I do it still has references
to IPA2.

So far I have tried the following.

 1. ipa-replica-manage del --force IPA2
 2. ipa-replica-manage del --force --cleanruv IPA2
 3. /usr/sbin/cleanallruv.pl 
 -D "cn=directory
manager" -w - -b "dc=example,dc=net" -r 6


Got the rid = 6 by running
ldapsearch -Y GSSAPI -b "dc=example,dc=net"

'(&(nsuniqueid=---)(objectclass=nstombstone))'
nsds50ruv

In the directory server logs, I guess its still trying to
connect to
IPA2 and failing. Below are some lines.

[06/Apr/2016:10:18:09 +] NSMMReplicationPlugin -
agmt="cn=meToipa2.example.net

" (ipa2:389):
Replication bind with GSSAPI auth failed: LDAP error -1
(Can't contact
LDAP server) ()
[06/Apr/2016:10:18:09 +] NSMMReplicationPlugin -
CleanAllRUV Task
(rid 6): Replica not online (agmt="cn=meToipa2.example.net

" (ipa2:389))
[06/Apr/2016:10:18:09 +] NSMMReplicationPlugin -
CleanAllRUV Task
(rid 6): Not all replicas online, retrying in 2560 seconds...

Any pointers would be helpful.


On ipa1 run:

% ipa-replica-manage list -v `hostname`

This will give the list of actual agreements and their status.

rob







--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael 
O'Neill

-- 
Manage your subscription 

Re: [Freeipa-users] start and stop of ipa commands in systemd

[Ludwig Krispenz]

On 04/04/2016 01:40 PM, Martin (Lists) wrote:

Am 04.04.2016 um 09:06 schrieb Martin Babinsky:

On 04/01/2016 08:53 PM, Martin (Lists) wrote:

Hallo

I have a question regarding enabling/disabling separate ipa parts in
systemd. Is it necessarry or required to have httpd, directory server,
named memcache and all the other ipa services to be enabled in systemd?
Or is it recomended to have only the main ipa service enabled (and all
the other disabled)?

Regards
Martin


Hi Martin,

ipa.service actually calls `ipactl` command which starts/stops all
individual components at once (dirsrv, http, kdc, kpasswd, memcache,
pki-tomcat etc.). All of these services (which are listed in `ipactl
status`) must be up and running for IPA server to work correctly in all
aspects.

So in this sense 'ipa.service' is just an umbrella that groups all the
components of FreeIPA installation.


For starting and stopping all neccessarry parts this is OK. But if I
have enabled some of these services directly in systemd (lets say
memcached or the ldap server) does that make problems during startup or
shutdown.

May be it is just a coincidence, but I had several warnings (up to
thousands) in the past from the LDAP Server at a simple restart of the
server:

DSRetroclPlugin - delete_changerecord: could not delete change record
553423 (rc: 32): 1 Time(s)

An I have not found any reason for this. Therefore the question: can
this be due to a false shutdown or startup sequence by systemd?
The DSRetroclPlugin messages occur when the starting point for trimming 
the retro changelog was incorrectly set. The messages themselves are 
harmless, just skipping no longer existing changes.
I think a crash or kill at shutdown will increase the probabilty to run 
into these scenarios


Last time I run "ipactl stop" before restarting the server and had no
such warnings. As I said may be its just a coincidence.

I run ipa on a up to date fedora 23 server.

Regards
Martin



--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael 
O'Neill

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] ipa replica failed PR_DeleteSemaphore

[Ludwig Krispenz]

On 03/14/2016 05:33 PM, Andrew E. Bruno wrote:

On Mon, Mar 14, 2016 at 09:35:15AM +0100, Ludwig Krispenz wrote:

On 03/12/2016 04:02 PM, Andrew E. Bruno wrote:

On Wed, Mar 09, 2016 at 06:08:04PM +0100, Ludwig Krispenz wrote:

On 03/09/2016 05:51 PM, Andrew E. Bruno wrote:

On Wed, Mar 09, 2016 at 05:21:50PM +0100, Ludwig Krispenz wrote:

[09/Mar/2016:11:33:03 -0500] NSMMReplicationPlugin - changelog program - 
_cl5NewDBFile: PR_DeleteSemaphore: 
/var/lib/dirsrv/slapd-CBLS-CCR-BUFFALO-EDU/cldb/ed35d212-2cb811e5-af63d574-de3f6355.sema;
 NSPR error - -5943

if ds is cleanly shutdown this file should be removed, if ds is killed it
remains and should be recreated at restart, which fails. could you try
another stop, remove the file manually and start again ?

We had our replicas crash again. Curious if it's safe to delete the
other db files as well:

ls -alh /var/lib/dirsrv/slapd-CBLS-CCR-BUFFALO-EDU/cldb/
   30  DBVERSION
6.8G  ed35d212-2cb811e5-af63d574-de3f6355_55a955910004.db
0  ed35d212-2cb811e5-af63d574-de3f6355.sema
  18M  f32bb356-2cb811e5-af63d574-de3f6355_55a955ca0060.db
0  f32bb356-2cb811e5-af63d574-de3f6355.sema


Should all these files be deleted if the ds is cleanly shutdown? or should we
only remove the *.sema files.

the *.db file contains the data of the changelog, if you delete them you
start with a new cl and could get into replication problems requiring
reinitialization. you normally shoul not delete them.
The .sema is used to control how many threads can concurrently access the
cl, it should be recreated at restart, so it is safe to delete them after a
crash.

Sounds good..thanks. We deleted the .sema files after the crash and the
replicas came back up ok.


If you getting frequent crashes, we shoul try to find the reason for the
crashes, could you try to get a core file ?

This time we had two replicas crash and ns-slapd wasn't running so we
couldn't grab a pstack. Here's a snip from the error logs right before
the crash (not sure if this is related or not):

[11/Mar/2016:09:57:56 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No 
original_tombstone for changenumber=11573832,cn=changelog!!
[11/Mar/2016:09:57:57 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No 
original_tombstone for changenumber=11575824,cn=changelog!!
[11/Mar/2016:09:57:58 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No 
original_tombstone for changenumber=11575851,cn=changelog!!
[11/Mar/2016:10:00:28 -0500] - libdb: BDB2055 Lock table is out of available 
lock entries
[11/Mar/2016:10:00:28 -0500] NSMMReplicationPlugin - changelog program - 
_cl5CompactDBs: failed to compact 986efe12-71b811e5-9d33a516-e778e883; db error 
- 12 Cannot allocate memory
[11/Mar/2016:10:02:07 -0500] - libdb: BDB2055 Lock table is out of available 
lock entries
[11/Mar/2016:10:02:07 -0500] - compactdb: failed to compact changelog; db error 
- 12 Cannot allocate memory
don't know if this is related to your crashes, but compation of 
changelog was running, probably for some time, and finally failed. The 
idea behind compaction is to compact a fragmented btree and reclaim some 
space, but it uses a transaction for the complete operation and lock 
every page accessed. This can be time consuming, blocking other txns, 
and run out of locks.


There are two options to address this, either increase the number of 
configured db locks (problem is there is no good hint how much locks 
will be needed), or disable changelog compaction, by setting:

dn: cn=changelog5,cn=config
..
nsslapd-changelogcompactdb-interval: 0


I would disable compaction, I don't think there is much benefit (in my 
memory BDB compaction was slow and not very effective) and it is better 
to avoid the side effects

[11/Mar/2016:12:36:18 -0500] - slapd_poll(377) timed out
[11/Mar/2016:13:06:17 -0500] - slapd_poll(377) timed out

We just upgraded to ipa 4.2 centos 7.2 and if we see anymore crashes
we'll try and get more info.

Thanks again.

--Andrew




--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael 
O'Neill

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] ipa replica failed PR_DeleteSemaphore

[Ludwig Krispenz]

On 03/12/2016 04:02 PM, Andrew E. Bruno wrote:

On Wed, Mar 09, 2016 at 06:08:04PM +0100, Ludwig Krispenz wrote:

On 03/09/2016 05:51 PM, Andrew E. Bruno wrote:

On Wed, Mar 09, 2016 at 05:21:50PM +0100, Ludwig Krispenz wrote:

[09/Mar/2016:11:33:03 -0500] NSMMReplicationPlugin - changelog program - 
_cl5NewDBFile: PR_DeleteSemaphore: 
/var/lib/dirsrv/slapd-CBLS-CCR-BUFFALO-EDU/cldb/ed35d212-2cb811e5-af63d574-de3f6355.sema;
 NSPR error - -5943

if ds is cleanly shutdown this file should be removed, if ds is killed it
remains and should be recreated at restart, which fails. could you try
another stop, remove the file manually and start again ?



We had our replicas crash again. Curious if it's safe to delete the
other db files as well:

ls -alh /var/lib/dirsrv/slapd-CBLS-CCR-BUFFALO-EDU/cldb/
   30  DBVERSION
6.8G  ed35d212-2cb811e5-af63d574-de3f6355_55a955910004.db
0  ed35d212-2cb811e5-af63d574-de3f6355.sema
  18M  f32bb356-2cb811e5-af63d574-de3f6355_55a955ca0060.db
0  f32bb356-2cb811e5-af63d574-de3f6355.sema


Should all these files be deleted if the ds is cleanly shutdown? or should we
only remove the *.sema files.
the *.db file contains the data of the changelog, if you delete them you 
start with a new cl and could get into replication problems requiring 
reinitialization. you normally shoul not delete them.
The .sema is used to control how many threads can concurrently access 
the cl, it should be recreated at restart, so it is safe to delete them 
after a crash.


If you getting frequent crashes, we shoul try to find the reason for the 
crashes, could you try to get a core file ?

http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes


Thanks,

--Andrew


--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael 
O'Neill

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] ipa replica failed PR_DeleteSemaphore

[Ludwig Krispenz]

On 03/09/2016 05:51 PM, Andrew E. Bruno wrote:

On Wed, Mar 09, 2016 at 05:21:50PM +0100, Ludwig Krispenz wrote:

On 03/09/2016 04:46 PM, Andrew E. Bruno wrote:

On Wed, Mar 09, 2016 at 10:37:05AM -0500, Andrew E. Bruno wrote:

On Wed, Mar 09, 2016 at 04:13:28PM +0100, Ludwig Krispenz wrote:

if the process hangs, could you get a pstack from the process ?

I'd be happy to provide a pstack but can't seem to get the correct debuginfo
packages installed.. we're running centos7 and  389-ds-base 1.3.3.1. We haven't
upgraded to 1.3.4.0. How can I get the debuginfo packages installed for that
specific version.

Nevermind.. i got the debuginfo packages. Attached is the stacktrace of
our second failed replicate that's currently hung.

not sure, but the process could be in a deadlock, there are threads in the
retro cl and memberof plugin and we have seen deadlocks there. In that case
restart or stop would not be able to stop the ds process. You can try ipactl
stop and if the ds process is still running, you have to kill it

Our first master came back up after the restart and appears to be
working again. The second replica that was hung, we did a ipactl stop
and it killed the ds process. Running a ipactl start now. We got the
same error:


[09/Mar/2016:11:33:03 -0500] NSMMReplicationPlugin - changelog program - 
_cl5NewDBFile: PR_DeleteSemaphore: 
/var/lib/dirsrv/slapd-CBLS-CCR-BUFFALO-EDU/cldb/ed35d212-2cb811e5-af63d574-de3f6355.sema;
 NSPR error - -5943
if ds is cleanly shutdown this file should be removed, if ds is killed 
it remains and should be recreated at restart, which fails. could you 
try another stop, remove the file manually and start again ?



We're going to let this run and hopefully will come back up.

Just want to confirm again that these can can be safely ignored:

[09/Mar/2016:10:23:10 -0500] DSRetroclPlugin - delete_changerecord: could not 
delete change record 11272989 (rc: 32)
[09/Mar/2016:10:23:10 -0500] DSRetroclPlugin - delete_changerecord: could not 
delete change record 11272990 (rc: 32)
there is something wrong with defining the starting point for changelog 
trimming, so it will skip many entries, this is annoying and we will 
have to fix it. Apart from spamming the logs and keeping the retor cl 
busy for a while it should not do any harm.


They fill up the logs when bringing the ds back up.

We seem to keep getting bit by this deadlock [1,2]. Replicas become
unresponsive, file descriptor counts increase. Other than a pstack, if there's
any other info we can provide/check let us know.

We'll be upgrading to centos 7.2 and 389-ds-base 1.3.4 next week.

As always, thanks again for the help and quick responses.

Best,

--Andrew

[1] https://www.redhat.com/archives/freeipa-users/2015-September/msg6.html
[2] https://www.redhat.com/archives/freeipa-users/2015-June/msg00389.html


--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael 
O'Neill

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] ipa replica failed PR_DeleteSemaphore

[Ludwig Krispenz]

On 03/09/2016 03:46 PM, Andrew E. Bruno wrote:

Hello,

We had a replica fail today with:

[09/Mar/2016:09:39:59 -0500] NSMMReplicationPlugin - changelog program - 
_cl5NewDBFile: PR_DeleteSemaphore: 
/var/lib/dirsrv/slapd-CBLS-CCR-BUFFALO-EDU/cldb/e909b405-2cb811e5-ac0b8f7e-e0b1a377.sema;
 NSPR error - -5943

the nspr error means:
/* Cannot create or rename a filename that already exists */
#define PR_FILE_EXISTS_ERROR (-5943L)

could you check if the file exists and if there is a permission problem 
for the dirsrv user to recreate it ?

if the process hangs, could you get a pstack from the process ?



dirsrv just hangs here. Doesn't seem to want to start up.. any pointers on 
where how to debug?

This is our first master so want to try and save it if possible.

Thanks,

--Andrew



--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael 
O'Neill

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] lock table errors

[Ludwig Krispenz]

On 02/23/2016 05:10 PM, Andy Thompson wrote:

On 02/23/2016 03:02 PM, Andy Thompson wrote:

Came across one of my replicas this morning with the following in
the error log

[20/Feb/2016:17:23:38 -0500] - libdb: BDB2055 Lock table is out of
available lock entries
[20/Feb/2016:17:23:38 -0500] entryrdn-index - _entryrdn_delete_key:
Deleting C1 failed; Cannot allocate memory(12)
[20/Feb/2016:17:23:38 -0500] - database index operation failed BAD
1031, err=12 Cannot allocate memory
[20/Feb/2016:17:23:38 -0500] -
index_del_entry(changenumber=1328662,cn=changelog, 0x26) failed

(12)

[20/Feb/2016:17:23:38 -0500] DSRetroclPlugin - delete_changerecord:
could not delete change record 1328662 (rc: 1)
[20/Feb/2016:17:23:38 -0500] - libdb: BDB2055 Lock table is out of
available lock entries
[20/Feb/2016:17:23:38 -0500] entryrdn-index - _entryrdn_get_elem:
Failed to position cursor at the key: 1328666: Cannot allocate
memory(12)
[20/Feb/2016:17:23:38 -0500] entryrdn-index - _entryrdn_delete_key:
Failed to position cursor at the key: 1328666: Cannot allocate
memory(12)
[20/Feb/2016:17:23:38 -0500] - libdb: BDB2055 Lock table is out of
available lock entries
[20/Feb/2016:17:23:38 -0500] - database index operation failed BAD
1031, err=12 Cannot allocate memory
[20/Feb/2016:17:23:38 -0500] -
index_del_entry(changenumber=1328663,cn=changelog, 0x26) failed

(12)

[20/Feb/2016:17:23:38 -0500] NSMMReplicationPlugin - changelog
program
- _cl5CompactDBs: failed to compact
5f1d2b12-cf1411e4-b055ba8a-f4b484f7; db error - 12 Cannot allocate
memory
[20/Feb/2016:17:23:38 -0500] DSRetroclPlugin - delete_changerecord:
could not delete change record 1328663 (rc: 1)
[20/Feb/2016:17:23:41 -0500] ldbm_back_delete - conn=0 op=0 [retry:
1]

No original_tombstone for changenumber=1330335,cn=changelog!!

And then nothing.  Was troubleshooting some clients that were having

issues resolving some trusted domain users.

I restarted IPA and it rolled through a few thousand missing change
records

23/Feb/2016:08:39:34 -0500] DSRetroclPlugin - delete_changerecord:
could not delete change record 1328696 (rc: 32)

Any thoughts as to what might have caused the lock table errors?

in BerkeleyDB this means that the number of pages which would have to
be locked in one transaction exceeds the configured number of locks.
This could happen if eg a large group is deleted and for each member
of the group inside the same transaction the memberof attribute has
to be modified

Are there any configuration options to increase that setting?  And would it

have caused the replica to become unresponsive?
you can change

nsslapd-db-locks

in the entry:

dn: cn=config,cn=ldbm database,cn=plugins,cn=config

yes. in that state it would not process updates, the txn should be finally
aborted and the system should recover,but ..

Is there any rule of thumb or anything I can look at to get an idea of what I 
should increase that to or should it even be necessary?

The current setting has a default of 1

cn=database,cn=monitor,cn=ldbm database,cn=plugins,cn=config

currently shows

nsslapd-db-current-locks: 82

What might cause that to spike up that significantly to deplete the locks?  
That's a pretty huge jump.
I have given you an example of what operation could use a high number of 
page locks, to find out what was going on in your case would require to 
investigate which operations were active when the problem started, what 
the entries modified added looked like ..


Running 389-ds-base-1.3.4.0-26.el7_2.x86_64

-andy



--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] lock table errors

[Ludwig Krispenz]

On 02/23/2016 03:43 PM, Andy Thompson wrote:

-Original Message-
From: freeipa-users-boun...@redhat.com [mailto:freeipa-users-
boun...@redhat.com] On Behalf Of Ludwig Krispenz
Sent: Tuesday, February 23, 2016 9:31 AM
To: freeipa-users@redhat.com
Subject: Re: [Freeipa-users] lock table errors


On 02/23/2016 03:02 PM, Andy Thompson wrote:

Came across one of my replicas this morning with the following in the
error log

[20/Feb/2016:17:23:38 -0500] - libdb: BDB2055 Lock table is out of
available lock entries
[20/Feb/2016:17:23:38 -0500] entryrdn-index - _entryrdn_delete_key:
Deleting C1 failed; Cannot allocate memory(12)
[20/Feb/2016:17:23:38 -0500] - database index operation failed BAD
1031, err=12 Cannot allocate memory
[20/Feb/2016:17:23:38 -0500] -
index_del_entry(changenumber=1328662,cn=changelog, 0x26) failed (12)
[20/Feb/2016:17:23:38 -0500] DSRetroclPlugin - delete_changerecord:
could not delete change record 1328662 (rc: 1)
[20/Feb/2016:17:23:38 -0500] - libdb: BDB2055 Lock table is out of
available lock entries
[20/Feb/2016:17:23:38 -0500] entryrdn-index - _entryrdn_get_elem:
Failed to position cursor at the key: 1328666: Cannot allocate
memory(12)
[20/Feb/2016:17:23:38 -0500] entryrdn-index - _entryrdn_delete_key:
Failed to position cursor at the key: 1328666: Cannot allocate
memory(12)
[20/Feb/2016:17:23:38 -0500] - libdb: BDB2055 Lock table is out of
available lock entries
[20/Feb/2016:17:23:38 -0500] - database index operation failed BAD
1031, err=12 Cannot allocate memory
[20/Feb/2016:17:23:38 -0500] -
index_del_entry(changenumber=1328663,cn=changelog, 0x26) failed (12)
[20/Feb/2016:17:23:38 -0500] NSMMReplicationPlugin - changelog program
- _cl5CompactDBs: failed to compact
5f1d2b12-cf1411e4-b055ba8a-f4b484f7; db error - 12 Cannot allocate
memory
[20/Feb/2016:17:23:38 -0500] DSRetroclPlugin - delete_changerecord:
could not delete change record 1328663 (rc: 1)
[20/Feb/2016:17:23:41 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1]

No original_tombstone for changenumber=1330335,cn=changelog!!

And then nothing.  Was troubleshooting some clients that were having

issues resolving some trusted domain users.

I restarted IPA and it rolled through a few thousand missing change
records

23/Feb/2016:08:39:34 -0500] DSRetroclPlugin - delete_changerecord:
could not delete change record 1328696 (rc: 32)

Any thoughts as to what might have caused the lock table errors?

in BerkeleyDB this means that the number of pages which would have to be
locked in one transaction exceeds the configured number of locks. This could
happen if eg a large group is deleted and for each member of the group
inside the same transaction the memberof attribute has to be modified


Are there any configuration options to increase that setting?  And would it 
have caused the replica to become unresponsive?

you can change

nsslapd-db-locks

in the entry:

dn: cn=config,cn=ldbm database,cn=plugins,cn=config

yes. in that state it would not process updates, the txn should be 
finally aborted and the system should recover,but ..


--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] lock table errors

[Ludwig Krispenz]

On 02/23/2016 03:02 PM, Andy Thompson wrote:

Came across one of my replicas this morning with the following in the error log

[20/Feb/2016:17:23:38 -0500] - libdb: BDB2055 Lock table is out of available 
lock entries
[20/Feb/2016:17:23:38 -0500] entryrdn-index - _entryrdn_delete_key: Deleting C1 
failed; Cannot allocate memory(12)
[20/Feb/2016:17:23:38 -0500] - database index operation failed BAD 1031, err=12 
Cannot allocate memory
[20/Feb/2016:17:23:38 -0500] - 
index_del_entry(changenumber=1328662,cn=changelog, 0x26) failed (12)
[20/Feb/2016:17:23:38 -0500] DSRetroclPlugin - delete_changerecord: could not 
delete change record 1328662 (rc: 1)
[20/Feb/2016:17:23:38 -0500] - libdb: BDB2055 Lock table is out of available 
lock entries
[20/Feb/2016:17:23:38 -0500] entryrdn-index - _entryrdn_get_elem: Failed to 
position cursor at the key: 1328666: Cannot allocate memory(12)
[20/Feb/2016:17:23:38 -0500] entryrdn-index - _entryrdn_delete_key: Failed to 
position cursor at the key: 1328666: Cannot allocate memory(12)
[20/Feb/2016:17:23:38 -0500] - libdb: BDB2055 Lock table is out of available 
lock entries
[20/Feb/2016:17:23:38 -0500] - database index operation failed BAD 1031, err=12 
Cannot allocate memory
[20/Feb/2016:17:23:38 -0500] - 
index_del_entry(changenumber=1328663,cn=changelog, 0x26) failed (12)
[20/Feb/2016:17:23:38 -0500] NSMMReplicationPlugin - changelog program - 
_cl5CompactDBs: failed to compact 5f1d2b12-cf1411e4-b055ba8a-f4b484f7; db error 
- 12 Cannot allocate memory
[20/Feb/2016:17:23:38 -0500] DSRetroclPlugin - delete_changerecord: could not 
delete change record 1328663 (rc: 1)
[20/Feb/2016:17:23:41 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No 
original_tombstone for changenumber=1330335,cn=changelog!!

And then nothing.  Was troubleshooting some clients that were having issues 
resolving some trusted domain users.

I restarted IPA and it rolled through a few thousand missing change records

23/Feb/2016:08:39:34 -0500] DSRetroclPlugin - delete_changerecord: could not 
delete change record 1328696 (rc: 32)

Any thoughts as to what might have caused the lock table errors?
in BerkeleyDB this means that the number of pages which would have to be 
locked in one transaction exceeds the configured number of locks. This 
could happen if eg a large group is deleted and for each member of the 
group inside the same transaction the memberof attribute has to be modified


Thanks

-andy



*** This communication may contain privileged and/or confidential information. 
It is intended solely for the use of the addressee. If you are not the intended 
recipient, you are strictly prohibited from disclosing, copying, distributing 
or using any of this information. If you received this communication in error, 
please contact the sender immediately and destroy the material in its entirety, 
whether electronic or hard copy. ***




--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

[Ludwig Krispenz]

On 02/22/2016 11:51 PM, Timothy Geier wrote:


What’s the established procedure to start a 389 instance without any 
replication agreements enabled?  The only thing that seemed close on 
google 
(http://directory.fedoraproject.org/docs/389ds/howto/howto-fix-and-reset-time-skew.html) 
seems risky and couldn’t be done

trivially in a production environment.
no, this is about how to get out of problems when replication could no 
longer synchronize its csn time generation, either by too many 
accumulate time drifts o playing with system time, hope you don't have 
to go thru this.


Enabling disabling a replication agreement can be done by setting the 
configuration parameter:


look for replication agreements (entries with 
objectclass=nsDS5ReplicationAgreement) and set

nsds5ReplicaEnabled: off

you can do this with an ldapmodify when the server is running or by 
editing /etc/dirsrv/slapd-/dse.ldif when teh server is stopped


--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

[Ludwig Krispenz]

The crash is an abort because of a failed assertion in  the kerberos code

Thread 1 (Thread 0x7fa7d4c88700 (LWP 3125)):
#0  0x7fa7e6ace5f7 in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x7fa7e6acfce8 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x7fa7e6ac7566 in __assert_fail_base () from /lib64/libc.so.6
No symbol table info available.
#3  0x7fa7e6ac7612 in __assert_fail () from /lib64/libc.so.6
No symbol table info available.
#4  0x7fa7e8d71b83 in k5_mutex_lock.part.1 () from /lib64/libkrb5.so.3
No symbol table info available.
#5  0x7fa7e8d7bda1 in k5_cc_mutex_lock () from /lib64/libkrb5.so.3
No symbol table info available.
#6  0x7fa7e8d851bf in krb5_mcc_store () from /lib64/libkrb5.so.3
No symbol table info available.
#7  0x7fa7e8d88070 in krb5_cc_store_cred () from /lib64/libkrb5.so.3
No symbol table info available.
#8  0x7fa7e8d98be3 in complete () from /lib64/libkrb5.so.3
No symbol table info available.
#9  0x7fa7e8d99ba1 in krb5_tkt_creds_step () from /lib64/libkrb5.so.3
No symbol table info available.
#10 0x7fa7e8d9a637 in krb5_tkt_creds_get () from /lib64/libkrb5.so.3
No symbol table info available.
#11 0x7fa7e8d9a75c in krb5_get_credentials () from /lib64/libkrb5.so.3
No symbol table info available.
#12 0x7fa7e0f6736a in krb5_gss_init_sec_context_ext () from 
/lib64/libgssapi_krb5.so.2

No symbol table info available.
#13 0x7fa7e0f67c97 in krb5_gss_init_sec_context () from 
/lib64/libgssapi_krb5.so.2

No symbol table info available.
#14 0x7fa7e0f516ab in gss_init_sec_context () from 
/lib64/libgssapi_krb5.so.2

No symbol table info available.
#15 0x7fa7e118ac44 in gssapi_client_mech_step () from 
/usr/lib64/sasl2/libgssapiv2.so

No symbol table info available.
#16 0x7fa7e72847f5 in sasl_client_step () from /lib64/libsasl2.so.3
No symbol table info available.
#17 0x7fa7e7284b76 in sasl_client_start () from /lib64/libsasl2.so.3
No symbol table info available.
#18 0x7fa7e8475e73 in ldap_int_sasl_bind () from 
/lib64/libldap_r-2.4.so.2

No symbol table info available.
#19 0x7fa7e8479492 in ldap_sasl_interactive_bind () from 
/lib64/libldap_r-2.4.so.2

No symbol table info available.
#20 0x7fa7e84796bd in ldap_sasl_interactive_bind_s () from 
/lib64/libldap_r-2.4.so.2


Directory server tries to open a replication connetion usig GSSAPI. I 
don't know which assertion fails in krb, but

- could you try with the replication agreement disabled ?
- Rob, you have been discussing renewals of keytabs, would we have to 
renew the ds.keytab ?



On 02/20/2016 07:57 AM, Timothy Geier wrote:

To update this thread..timeshifting to before the certs expired (Feb.
2) brings up the interesting issue of pki-tomcatd (sometimes) starting
while slapd crashes almost immediately after ipactl start is run..if it
doesn't crash right away, pki-tomcatd will start, but since slapd will
then go down right afterwards, the certs cannot be renewed.  After
returning to real time, slapd is just fine along with everything else
aside from pki-tomcatd.

A core dump is attached.


"This message and any attachments may contain confidential information. If you
have received this  message in error, any use or distribution is prohibited.
Please notify us by reply e-mail if you have mistakenly received this message,
and immediately and permanently delete it and any attachments. Thank you."




-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Failed to setup replica, slapi_ldap_bind fails

[Ludwig Krispenz]

On 02/12/2016 06:22 PM, Filip Pytloun wrote:

Following is in /etc/ldap/ldap.conf on both servers (only URI differs):

what is your OS, do you also have a /etc/openldap/ldap.conf

ldapsearch and the replication connection shoudl use the same openldap 
libraries and so it is strange that -ZZ works and indside ds doesn't.


At what point did your replica install fail, is there any hint in the 
replica install log ?


TLS_CACERT /etc/ipa/ca.crt
TLS_REQCERT allow
URI ldaps://idm02.tcpcloud.eu
BASE dc=tcpcloud,dc=eu

As ldapsearch is passing just fine on both nodes, I don't suppose
ldap.conf is wrong.
I also tried to set TLS_REQCERT to allow just to be sure (in case that
bad cert is provided).

On 2016/02/12 16:57, Ludwig Krispenz wrote:

On 02/12/2016 03:35 PM, Filip Pytloun wrote:

It's the same as for idm01:

[12/Feb/2016:15:24:26 +0100] NSMMReplicationPlugin - 
agmt="cn=meToidm01.tcpcloud.eu" (idm01:389): Replication bind with SIMPLE auth 
failed: LDAP error -11 (Connect error) ((unknown error code))
[12/Feb/2016:15:24:27 +0100] slapi_ldap_bind - Error: could not send startTLS 
request: error -11 (Connect error) errno 0 (Success)

you can get this connect error if the client side cannot verify the cert the
server sends, could you check what you have in f


In access logs I can't read much interesting, just that TLS connection happened 
from idm01:

[12/Feb/2016:15:33:11 +0100] conn=14 fd=64 slot=64 connection from 185.22.97.19 
to 172.10.10.192
[12/Feb/2016:15:33:11 +0100] conn=14 op=0 EXT oid="1.3.6.1.4.1.1466.20037" 
name="startTLS"
[12/Feb/2016:15:33:11 +0100] conn=14 op=0 RESULT err=0 tag=120 nentries=0 
etime=0
[12/Feb/2016:15:33:11 +0100] conn=14 TLS1.2 128-bit AES-GCM
[12/Feb/2016:15:33:11 +0100] conn=14 op=-1 fd=64 closed - B1
[12/Feb/2016:15:33:59 +0100] conn=15 fd=64 slot=64 connection from 185.22.97.19 
to 172.10.10.192
[12/Feb/2016:15:33:59 +0100] conn=15 op=0 EXT oid="1.3.6.1.4.1.1466.20037" 
name="startTLS"
[12/Feb/2016:15:33:59 +0100] conn=15 op=0 RESULT err=0 tag=120 nentries=0 
etime=0
[12/Feb/2016:15:34:00 +0100] conn=15 TLS1.2 128-bit AES-GCM
[12/Feb/2016:15:34:00 +0100] conn=15 op=-1 fd=64 closed - B1

On 2016/02/12 15:22, Ludwig Krispenz wrote:

On 02/12/2016 03:06 PM, Filip Pytloun wrote:

Hello,

even when enabling replication logging, I get nothing useful in logs:

[12/Feb/2016:14:57:00 +0100] NSMMReplicationPlugin - 
agmt="cn=meToidm02.tcpcloud.eu" (idm02:389): Trying secure startTLS 
slapi_ldap_init_ext
[12/Feb/2016:14:57:00 +0100] NSMMReplicationPlugin - 
agmt="cn=meToidm02.tcpcloud.eu" (idm02:389): binddn = cn=replication 
manager,cn=config,  passwd = {AES-some_encrypted_password
[12/Feb/2016:14:57:01 +0100] slapi_ldap_bind - Error: could not send startTLS 
request: error -11 (Connect error) errno 0 (Success)
[12/Feb/2016:14:57:01 +0100] NSMMReplicationPlugin - 
agmt="cn=meToidm02.tcpcloud.eu" (idm02:389): Replication bind with SIMPLE auth 
failed: LDAP error -11 (Connect error) ((unknown error code))
[12/Feb/2016:14:57:01 +0100] NSMMReplicationPlugin - 
agmt="cn=meToidm02.tcpcloud.eu" (idm02:389): Disconnected from the consumer

what is in the access and error logs of idm02 for this time ?

But I can bind just fine manually:

ldapsearch -D "cn=replication manager,cn=config" -w some_password -b cn=config 
-h idm02 -ZZ

I am starting to be clueless, nobody has an idea what could be wrong?

- DNS including PTR records are set up fine
- /etc/hosts is setup fine
- conncheck passes fine between nodes
- I can bind manually just fine

On 2016/02/08 18:05, Filip Pytloun wrote:

Hello,

I have a weird issue setting up FreeIPA replica. Conncheck passes fine
but at the end of ipa-replica-install I always get following error:

slapi_ldap_bind -Error: could not send startTLS request: error -11
(Connect error) errno 0 (Success)

on both master and replica without any further explanation in logs.

/etc/ldap.conf is correctly setup before ipa-replica-install and IPA CA
certificate is installed in system CA bundle so TLS should work just
fine.

Also I can manually connect just fine from replica to master and back so
it's not a network or LDAP client issue.

Replica agreement looks like this: http://pastebin.com/FT3p3KUk

freeipa-server 4.1.4
389-ds 1.3.4.5

Has anyone idea where to look at?

Filip



--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project



--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project


--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Failed to setup replica, slapi_ldap_bind fails

[Ludwig Krispenz]

On 02/12/2016 03:06 PM, Filip Pytloun wrote:

Hello,

even when enabling replication logging, I get nothing useful in logs:

[12/Feb/2016:14:57:00 +0100] NSMMReplicationPlugin - 
agmt="cn=meToidm02.tcpcloud.eu" (idm02:389): Trying secure startTLS 
slapi_ldap_init_ext
[12/Feb/2016:14:57:00 +0100] NSMMReplicationPlugin - 
agmt="cn=meToidm02.tcpcloud.eu" (idm02:389): binddn = cn=replication 
manager,cn=config,  passwd = {AES-some_encrypted_password
[12/Feb/2016:14:57:01 +0100] slapi_ldap_bind - Error: could not send startTLS 
request: error -11 (Connect error) errno 0 (Success)
[12/Feb/2016:14:57:01 +0100] NSMMReplicationPlugin - 
agmt="cn=meToidm02.tcpcloud.eu" (idm02:389): Replication bind with SIMPLE auth 
failed: LDAP error -11 (Connect error) ((unknown error code))
[12/Feb/2016:14:57:01 +0100] NSMMReplicationPlugin - 
agmt="cn=meToidm02.tcpcloud.eu" (idm02:389): Disconnected from the consumer

what is in the access and error logs of idm02 for this time ?


But I can bind just fine manually:

ldapsearch -D "cn=replication manager,cn=config" -w some_password -b cn=config 
-h idm02 -ZZ

I am starting to be clueless, nobody has an idea what could be wrong?

- DNS including PTR records are set up fine
- /etc/hosts is setup fine
- conncheck passes fine between nodes
- I can bind manually just fine

On 2016/02/08 18:05, Filip Pytloun wrote:

Hello,

I have a weird issue setting up FreeIPA replica. Conncheck passes fine
but at the end of ipa-replica-install I always get following error:

slapi_ldap_bind -Error: could not send startTLS request: error -11
(Connect error) errno 0 (Success)

on both master and replica without any further explanation in logs.

/etc/ldap.conf is correctly setup before ipa-replica-install and IPA CA
certificate is installed in system CA bundle so TLS should work just
fine.

Also I can manually connect just fine from replica to master and back so
it's not a network or LDAP client issue.

Replica agreement looks like this: http://pastebin.com/FT3p3KUk

freeipa-server 4.1.4
389-ds 1.3.4.5

Has anyone idea where to look at?

Filip






-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Failed to setup replica, slapi_ldap_bind fails

[Ludwig Krispenz]

On 02/12/2016 03:35 PM, Filip Pytloun wrote:

It's the same as for idm01:

[12/Feb/2016:15:24:26 +0100] NSMMReplicationPlugin - 
agmt="cn=meToidm01.tcpcloud.eu" (idm01:389): Replication bind with SIMPLE auth 
failed: LDAP error -11 (Connect error) ((unknown error code))
[12/Feb/2016:15:24:27 +0100] slapi_ldap_bind - Error: could not send startTLS 
request: error -11 (Connect error) errno 0 (Success)
you can get this connect error if the client side cannot verify the cert 
the server sends, could you check what you have in /etc/openldap/ldap.conf




In access logs I can't read much interesting, just that TLS connection happened 
from idm01:

[12/Feb/2016:15:33:11 +0100] conn=14 fd=64 slot=64 connection from 185.22.97.19 
to 172.10.10.192
[12/Feb/2016:15:33:11 +0100] conn=14 op=0 EXT oid="1.3.6.1.4.1.1466.20037" 
name="startTLS"
[12/Feb/2016:15:33:11 +0100] conn=14 op=0 RESULT err=0 tag=120 nentries=0 
etime=0
[12/Feb/2016:15:33:11 +0100] conn=14 TLS1.2 128-bit AES-GCM
[12/Feb/2016:15:33:11 +0100] conn=14 op=-1 fd=64 closed - B1
[12/Feb/2016:15:33:59 +0100] conn=15 fd=64 slot=64 connection from 185.22.97.19 
to 172.10.10.192
[12/Feb/2016:15:33:59 +0100] conn=15 op=0 EXT oid="1.3.6.1.4.1.1466.20037" 
name="startTLS"
[12/Feb/2016:15:33:59 +0100] conn=15 op=0 RESULT err=0 tag=120 nentries=0 
etime=0
[12/Feb/2016:15:34:00 +0100] conn=15 TLS1.2 128-bit AES-GCM
[12/Feb/2016:15:34:00 +0100] conn=15 op=-1 fd=64 closed - B1

On 2016/02/12 15:22, Ludwig Krispenz wrote:

On 02/12/2016 03:06 PM, Filip Pytloun wrote:

Hello,

even when enabling replication logging, I get nothing useful in logs:

[12/Feb/2016:14:57:00 +0100] NSMMReplicationPlugin - 
agmt="cn=meToidm02.tcpcloud.eu" (idm02:389): Trying secure startTLS 
slapi_ldap_init_ext
[12/Feb/2016:14:57:00 +0100] NSMMReplicationPlugin - 
agmt="cn=meToidm02.tcpcloud.eu" (idm02:389): binddn = cn=replication 
manager,cn=config,  passwd = {AES-some_encrypted_password
[12/Feb/2016:14:57:01 +0100] slapi_ldap_bind - Error: could not send startTLS 
request: error -11 (Connect error) errno 0 (Success)
[12/Feb/2016:14:57:01 +0100] NSMMReplicationPlugin - 
agmt="cn=meToidm02.tcpcloud.eu" (idm02:389): Replication bind with SIMPLE auth 
failed: LDAP error -11 (Connect error) ((unknown error code))
[12/Feb/2016:14:57:01 +0100] NSMMReplicationPlugin - 
agmt="cn=meToidm02.tcpcloud.eu" (idm02:389): Disconnected from the consumer

what is in the access and error logs of idm02 for this time ?

But I can bind just fine manually:

ldapsearch -D "cn=replication manager,cn=config" -w some_password -b cn=config 
-h idm02 -ZZ

I am starting to be clueless, nobody has an idea what could be wrong?

- DNS including PTR records are set up fine
- /etc/hosts is setup fine
- conncheck passes fine between nodes
- I can bind manually just fine

On 2016/02/08 18:05, Filip Pytloun wrote:

Hello,

I have a weird issue setting up FreeIPA replica. Conncheck passes fine
but at the end of ipa-replica-install I always get following error:

slapi_ldap_bind -Error: could not send startTLS request: error -11
(Connect error) errno 0 (Success)

on both master and replica without any further explanation in logs.

/etc/ldap.conf is correctly setup before ipa-replica-install and IPA CA
certificate is installed in system CA bundle so TLS should work just
fine.

Also I can manually connect just fine from replica to master and back so
it's not a network or LDAP client issue.

Replica agreement looks like this: http://pastebin.com/FT3p3KUk

freeipa-server 4.1.4
389-ds 1.3.4.5

Has anyone idea where to look at?

Filip




--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project





-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Master Error with two Master CentOS 7.2

[Ludwig Krispenz]

On 01/26/2016 09:45 AM, Günther J. Niederwimmer wrote:

Hello List,

I set up a CentOS 7.2 System with two master Server now I found this 1000 x
Error on my first master?

attrlist_replace - attr_replace (nsslapd-referral, ldap://ipa1.xxx.at:389/
o%3Dipaca) failed.
did you install and reinstall the replica on the same machine ? The 
message is usually related to removed replicaid, which was not properly 
cleaned.


can you do some searches ?. On both masters check which is the replicaID 
in use and which are the known ruvs:
ldapsearch -b "cn=config"  "objectclass=nsds5replica" replicaid 
nsds50ruv



Ludwig


the second is harmless  I read ;-)
NSMMReplicationPlugin - replication keep alive entry 

Re: [Freeipa-users] Master Error with two Master CentOS 7.2

[Ludwig Krispenz]

On 01/26/2016 12:30 PM, Günther J. Niederwimmer wrote:

Hello Ludwig,

Am Dienstag, 26. Januar 2016, 11:03:27 CET schrieb Ludwig Krispenz:

On 01/26/2016 09:45 AM, Günther J. Niederwimmer wrote:

Hello List,

I set up a CentOS 7.2 System with two master Server now I found this 1000
x
Error on my first master?

attrlist_replace - attr_replace (nsslapd-referral,
ldap://ipa1.xxx.at:389/ o%3Dipaca) failed.

did you install and reinstall the replica on the same machine ? The
message is usually related to removed replicaid, which was not properly
cleaned.

Yes, I must delete and reinstall the Replica but I have all cleanup I found in
the DOC

ipa-replica-manage del ipa1..at
ipa-csreplica-manage del ipa1...at

and create a new

ipa-replica-prepare ipa1.xxx.at

the system for ipa1 is a new installed KVM guest., with the same name
ipa1..at
  

can you do some searches ?. On both masters check which is the replicaID
in use and which are the known ruvs:
ldapsearch -b "cn=config"  "objectclass=nsds5replica" replicaid
nsds50ruv

Please can you give me the full command I am not a professional  for LDAP
ldapsearch -LLL -o ldif-wrap=no -x -h  -p 389  -D "cn=directory 
manager" -W -b "cn=config" "objectclass=nsds5replica" nsds5replicaid 
nsds50ruv


for host insert your masters


Thanks


the second is harmless  I read ;-)
NSMMReplicationPlugin - replication keep alive entry 

Re: [Freeipa-users] Master Error with two Master CentOS 7.2

[Ludwig Krispenz]

Hi,

you got a replicaid (97) leftover form the previous install for the 
o=ipaca backend. The other backend is ok, ipa-replica-manage del did the 
cleanup, but ipa-csreplica-manage doesn't. So you have to clean it 
manually by an ldap command.


Execute the following mod on one of the servers:

ldapmodify -D "cn=directory manager" -W -a
dn: cn=clean 97, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: o=ipaca
replica-id: 97
cn: clean 97


Ludwig

On 01/26/2016 04:52 PM, Günther J. Niederwimmer wrote:

Hello Ludwig,

Am Dienstag, 26. Januar 2016, 14:48:31 CET schrieb Ludwig Krispenz:

On 01/26/2016 12:30 PM, Günther J. Niederwimmer wrote:

Am Dienstag, 26. Januar 2016, 11:03:27 CET schrieb Ludwig Krispenz:

On 01/26/2016 09:45 AM, Günther J. Niederwimmer wrote:

I set up a CentOS 7.2 System with two master Server now I found this
1000
x
Error on my first master?

attrlist_replace - attr_replace (nsslapd-referral,
ldap://ipa1.xxx.at:389/ o%3Dipaca) failed.

did you install and reinstall the replica on the same machine ? The
message is usually related to removed replicaid, which was not properly
cleaned.

Yes, I must delete and reinstall the Replica but I have all cleanup I
found in the DOC

ipa-replica-manage del ipa1..at
ipa-csreplica-manage del ipa1...at

and create a new

ipa-replica-prepare ipa1.xxx.at

the system for ipa1 is a new installed KVM guest., with the same name
ipa1..at


can you do some searches ?. On both masters check which is the replicaID
in use and which are the known ruvs:
ldapsearch -b "cn=config"  "objectclass=nsds5replica" replicaid
nsds50ruv

Please can you give me the full command I am not a professional  for LDAP

ldapsearch -LLL -o ldif-wrap=no -x -h  -p 389  -D "cn=directory
manager" -W -b "cn=config" "objectclass=nsds5replica" nsds5replicaid
nsds50ruv
  

for host insert your masters

Thanks for the help.

The original master

dn: cn=replica,cn=dc\3Desslmaier\2Cdc\3Dat,cn=mapping tree,cn=config
nsds5replicaid: 4
nsds50ruv: {replicageneration} 562f579c0004
nsds50ruv: {replica 4 ldap://ipa.esslmaier.at:389} 562f57b70004
56a792640004
nsds50ruv: {replica 5 ldap://ipa1.esslmaier.at:389} 568a1fa20005
56a5cf7300020005

dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nsds5replicaid: 96
nsds50ruv: {replicageneration} 562f57e30060
nsds50ruv: {replica 96 ldap://ipa.esslmaier.at:389} 562f58040060
56a790210060
nsds50ruv: {replica 91 ldap://ipa1.esslmaier.at:389} 568a1ff7005b
568a20250006005b
nsds50ruv: {replica 97 ldap://ipa1.esslmaier.at:389} 562f58110061
5630a9c40061

The first replica master.

nsds5replicaid: 5
nsds50ruv: {replicageneration} 562f579c0004
nsds50ruv: {replica 5 ldap://ipa1.esslmaier.at:389} 568a1fa20005
56a793fc0005
nsds50ruv: {replica 4 ldap://ipa.esslmaier.at:389} 562f57b70004
56a792640004

dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nsds5replicaid: 91
nsds50ruv: {replicageneration} 562f57e30060
nsds50ruv: {replica 91 ldap://ipa1.esslmaier.at:389} 568a1ff7005b
568a20250006005b
nsds50ruv: {replica 96 ldap://ipa.esslmaier.at:389} 562f58040060
56a793a50060
nsds50ruv: {replica 97 ldap://ipa1.esslmaier.at:389} 562f58110061
5630a9c40061



Is this a bad Error ?

Can I do anything

Thanks for a answer,




--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Replica Error with freeIPA Centos 7.2

[Ludwig Krispenz]

On 01/25/2016 01:43 PM, Martin Kosek wrote:

On 01/25/2016 01:34 PM, thierry bordaz wrote:

On 01/23/2016 11:08 PM, Günther J. Niederwimmer wrote:

Hello,

I have installed freeIPA from a CentOS 7.2 with a replica Server, but I have
on all two masters a Error.

NSMMReplicationPlugin - replication keep alive entry 

Re: [Freeipa-users] Replica Error with freeIPA Centos 7.2

[Ludwig Krispenz]

On 01/23/2016 11:08 PM, Günther J. Niederwimmer wrote:

Hello,

I have installed freeIPA from a CentOS 7.2 with a replica Server, but I have
on all two masters a Error.

NSMMReplicationPlugin - replication keep alive entry 

Re: [Freeipa-users] Incremental update failed and requires administrator action

[Ludwig Krispenz]

could you get a core dump from the crash:
http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes

Ludwig

On 01/25/2016 12:08 PM, bahan w wrote:

Hello !

I recently installed a replica (master2) in addition of my master 
(master1) with IPA 3.0.0-47 on RHEL6.6.
I don't know from when exactly, but the dirsrv (and the whole ipa 
service) on master1 crashes regularly with the following logs.


###
[22/Jan/2016:15:38:20 +0100] - 389-Directory/1.2.11.15 
 B2015.279.183 starting up
[22/Jan/2016:15:38:20 +0100] schema-compat-plugin - warning: no 
entries set up under cn=computers, cn=compat,dc=
[22/Jan/2016:15:38:21 +0100] schema-compat-plugin - warning: no 
entries set up under cn=ng, cn=compat,dc=
[22/Jan/2016:15:38:21 +0100] schema-compat-plugin - warning: no 
entries set up under ou=sudoers,dc=
[22/Jan/2016:15:38:21 +0100] - slapd started.  Listening on All 
Interfaces port 389 for LDAP requests
[22/Jan/2016:15:38:21 +0100] - Listening on All Interfaces port 636 
for LDAPS requests
[22/Jan/2016:15:38:21 +0100] - Listening on 
/var/run/slapd-.socket for LDAPI requests
[22/Jan/2016:17:04:03 +0100] NSMMReplicationPlugin - changelog program 
- _cl5WriteOperationTxn: retry (49) the transaction 
(csn=56a252ef0004) failed (rc=-30994 (DB_LOCK_DEADLOCK: Locker 
killed to resolve a deadlock))
[22/Jan/2016:17:04:03 +0100] NSMMReplicationPlugin - changelog program 
- _cl5WriteOperationTxn: failed to write entry with csn 
(56a252ef0004); db error - -30994 DB_LOCK_DEADLOCK: Locker 
killed to resolve a deadlock
[22/Jan/2016:17:04:03 +0100] NSMMReplicationPlugin - 
write_changelog_and_ruv: can't add a change for 
uid=,cn=users,cn=accounts,dc= (uniqid: 
a7ebd403-c12111e5-9c84c092-9a5deb81, optype: 16) to changelog csn 
56a252ef0004
[22/Jan/2016:17:04:03 +0100] NSMMReplicationPlugin - 
agmt="cn=meTo" (:389): Missing data 
encountered
[22/Jan/2016:17:04:03 +0100] NSMMReplicationPlugin - 
agmt="cn=meTo" (:389): Incremental update 
failed and requires administrator action

###

Then the dirsrv, I mean the whole ipa server, is down.
When I restart the service, here is what is see :

###
[22/Jan/2016:17:06:18 +0100] - 389-Directory/1.2.11.15 
 B2015.279.183 starting up
[22/Jan/2016:17:06:18 +0100] - Detected Disorderly Shutdown last time 
Directory Server was running, recovering database.
[22/Jan/2016:17:06:18 +0100] schema-compat-plugin - warning: no 
entries set up under cn=computers, cn=compat,dc=
[22/Jan/2016:17:06:19 +0100] schema-compat-plugin - warning: no 
entries set up under cn=ng, cn=compat,dc=
[22/Jan/2016:17:06:19 +0100] schema-compat-plugin - warning: no 
entries set up under ou=sudoers,dc=
[22/Jan/2016:17:06:20 +0100] set_krb5_creds - Could not get initial 
credentials for principal [ldap/@] in keytab 
[FILE:/etc/dirsrv/ds.keytab]: -1765328324 (Generic error (see e-text))
[22/Jan/2016:17:06:20 +0100] - slapd started.  Listening on All 
Interfaces port 389 for LDAP requests
[22/Jan/2016:17:06:20 +0100] - Listening on All Interfaces port 636 
for LDAPS requests
[22/Jan/2016:17:06:20 +0100] - Listening on 
/var/run/slapd-.socket for LDAPI requests
[22/Jan/2016:17:06:20 +0100] slapd_ldap_sasl_interactive_bind - Error: 
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 
-2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified 
GSS failure.  Minor code may provide more information (Credentials 
cache file '/tmp/krb5cc_244' not found)) errno 0 (Success)
[22/Jan/2016:17:06:20 +0100] slapi_ldap_bind - Error: could not 
perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
[22/Jan/2016:17:06:20 +0100] NSMMReplicationPlugin - 
agmt="cn=meTo" (:389): Replication bind 
with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): 
generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code 
may provide more information (Credentials cache file '/tmp/krb5cc_244' 
not found))
[22/Jan/2016:17:06:23 +0100] NSMMReplicationPlugin - 
agmt="cn=meTo" (:389): Replication bind 
with GSSAPI auth resumed

###

It seems that there is a problem to write an entry in the DB ? Do you 
know how I can solve this problem please ?


Furthermore, it seems that there is a second problem with the keytab 
/etc/dirsrv/ds.keytab.


The keytab is good for me :
###
#ls -l /etc/dirsrv/ds.keytab
-rw--- 1 dirsrv dirsrv 362 Jan 21 14:12 /etc/dirsrv/ds.keytab
# kinit -kt /etc/dirsrv/ds.keytab ldap/@
# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: ldap/@

Valid starting ExpiresService principal
01/25/16 11:54:23  01/26/16 11:54:23 krbtgt/@
###

I wonder if this second problem does not come from the user dirsrv who 
would not be able to use this keytab.

I cannot test this because this user dirsrv has been created with nologin.
###
# su - dirsrv -c "kinit -kt /etc/dirsrv/ds.keytab 
ldap/@"

This account is currently not available.

# grep dirsrv /etc/passwd

Re: [Freeipa-users] Freeipa 4.3.0 replica installation fails with DuplicateEntry: This entry already exists

[Ludwig Krispenz]

On 01/22/2016 04:48 AM, Nathan Peters wrote:

Here are the results for that aci search using a non gssapi bind by directory 
manager on the old master that we are attempting to join agains.  I don't see 
anything in this list that would indicate that some users should or should not 
have access through a certain method.  Unless one of those sasl config settings 
is doing it ?

[root@dc2-ipa-dev-nvan ~]# ldapsearch -D "cn=directory manager" -W -b "cn=config" 
"(aci=*)"

you need to request the aci attribute to see the acis:

ldapsearch -D "cn=directory manager" -W -b "cn=config" "(aci=*)" aci



Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base 

Re: [Freeipa-users] Freeipa 4.3.0 replica installation fails with DuplicateEntry: This entry already exists

[Ludwig Krispenz]

On 01/21/2016 08:50 AM, Nathan Peters wrote:

I don't know if this makes a difference too, but I performed the same checks on 
a different completely working and joined FreeIPA master, against other 
masters, and even against itself directly.

It seems that no account, no keytab, and no host can see that mapping tree 
branch no matter who they search from or against if GSSAPI is used.
there should be no difference in the result, it should only depend on 
the acis and in one of your previous posts you said that you don't get a 
result bound as admin:

>>>

[root@dc2-ipa-dev-van ~]# ldapsearch -Hldaps://dc2-ipa-dev-nvan.mydomain.net  -b 
"cn=replica,cn=dc\3Dmydomain\2Cdc\3Dnet,cn=mapping tree,cn=config" -D 
"uid=admin,cn=users,cn=accounts,dc=mydomain,dc=net" -W
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base 

Re: [Freeipa-users] ns-slapd using all CPU ressources

[Ludwig Krispenz]

Hi,
if you are running 389-ds 1.3.4+ you may hit, ticket #48379. It id fixed 
and a new build is in preparation


Ludwig

On 01/19/2016 03:39 PM, Domingues Luis Filipe wrote:

Hi,

Reading the backtrace I have 30 threads with the same stack:

Thread 6 (Thread 0x7f572efed700 (LWP 1335)):
#0  0x7f576f80a877 in sched_yield () from /lib64/libc.so.6
No symbol table info available.
#1  0x7f577014df28 in PR_Sleep () from /lib64/libnspr4.so
No symbol table info available.
#2  0x55c939e9e7c7 in connection_threadmain ()
No symbol table info available.
#3  0x7f577014d5cb in _pt_root () from /lib64/libnspr4.so
No symbol table info available.
#4  0x7f576faec60a in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x7f576f826a4d in clone () from /lib64/libc.so.6
No symbol table info available.

While the other instance which is running fine, almost all threads are waiting 
on a cond_wait, with thise stack:
Thread 48 (Thread 0x7fced53a9700 (LWP 1871)):
#0  0x7fcee9269b10 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
No symbol table info available.
#1  0x7fcee98bfcf0 in PR_WaitCondVar () from /lib64/libnspr4.so
No symbol table info available.
#2  0x7fceeb7172c8 in slapi_wait_condvar () from 
/usr/lib64/dirsrv/libslapd.so.0
No symbol table info available.
#3  0x7fcee127a67e in cos_cache_wait_on_change () from 
/usr/lib64/dirsrv/plugins/libcos-plugin.so
No symbol table info available.
#4  0x7fcee98c55cb in _pt_root () from /lib64/libnspr4.so
No symbol table info available.
#5  0x7fcee926460a in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6  0x7fcee8f9ea4d in clone () from /lib64/libc.so.6
No symbol table info available.

Luis.

From: Rob Crittenden [rcrit...@redhat.com]
Sent: Friday, January 15, 2016 3:51 PM
To: Domingues Luis Filipe; freeipa-users@redhat.com
Cc: Aviolat Romain
Subject: Re: [Freeipa-users] ns-slapd using all CPU ressources

Domingues Luis Filipe wrote:

Hi all,

On our infra, we have two machines running Fedora with FreeIPA installed.

we have an issue with ns-slapd using 100% of CPU after a while. If we
restart the service, it starts to use all CPU resources after one day.

Outpute of the command strace -c -p  running for 4 minutes is:

% time seconds  usecs/call callserrors syscall
-- --- --- - - 
  99.80  229.603633   11247 20415   poll
   0.150.340032  10 32983 4 futex
   0.050.114068  114068 1   restart_syscall
   0.000.003464   0 20420 20416 getpeername
   0.000.002752   0 20416   clock_gettime
   0.000.001920   0  9840   read
   0.000.000205   545   close
   0.000.36   222   access
   0.000.17   122   open
   0.000.16   124   accept
   0.000.12   045   setsockopt
   0.000.07   022   fstat
   0.000.00   022   stat
   0.000.00   0 1   sendto
   0.000.00   024   getsockname
   0.000.00   0 4   getsockopt
   0.000.00   070   fcntl
   0.000.00   022   gettimeofday
-- --- --- - - 
100.00  230.066162104398 20420 total



Plus we looked at the syscalls using FTrace:

ns-slapd-7963  [000]  4063846.395630: sys_sched_yield()
ns-slapd-7956  [000]  4063846.395631: sys_sched_yield -> 0x0
ns-slapd-7956  [000]  4063846.395632: sys_sched_yield()
ns-slapd-7973  [000]  4063846.395633: sys_sched_yield -> 0x0
ns-slapd-7973  [000]  4063846.395634: sys_sched_yield()
ns-slapd-7965  [000]  4063846.395635: sys_sched_yield -> 0x0
ns-slapd-7965  [000]  4063846.395637: sys_sched_yield()
ns-slapd-7963  [000]  4063846.395637: sys_sched_yield -> 0x0
ns-slapd-7963  [000]  4063846.395639: sys_sched_yield()
ns-slapd-7956  [000]  4063846.395640: sys_sched_yield -> 0x0
ns-slapd-7956  [000]  4063846.395641: sys_sched_yield()
ns-slapd-7973  [000]  4063846.395642: sys_sched_yield -> 0x0
ns-slapd-7973  [000]  4063846.395643: sys_sched_yield()
ns-slapd-7965  [000]  4063846.395644: sys_sched_yield -> 0x0

The sys_sched_yield function is called almost every 2 microseconds. It seems 
too much.

Your best bet is to get a pstack or full backtrace to see what 389-ds is
doing. See
http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-hangs

rob



--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to 

Re: [Freeipa-users] Freeipa 4.3.0 replica installation fails with DuplicateEntry: This entry already exists

[Ludwig Krispenz]

On 01/18/2016 04:47 AM, Nathan Peters wrote:


This is another issue I'm not sure how to debug or solve in 4.3.0.  A 
failed replica installation left a replica with stuff in the tree, but 
not configured properly on the localhost.  I did ipa-server-install 
--uninstall as suggested by the installation program and it deleted 
the local copy of the data, but did not clean the tree.


Now all subsequent installations are failing with some duplicate entry 
error.


All packages are up to date so this is not the pki-ca 10.2.6-13 fix 
issue.  I've checked the whole tree for any references to the old copy 
of the master but I can't find them.


That error log is typically unhelpful as it doesn't tell me what entry 
or where it is looking or finding a duplicate or I would just go 
delete it myself.



look at the DS access log, you should see an ADD operation with
RESULT  err=68 tag=105


2016-01-18T03:29:55Z DEBUG Fetching nsDS5ReplicaId from master 
[attempt 1/5]


2016-01-18T03:29:55Z DEBUG Successfully updated nsDS5ReplicaId.

2016-01-18T03:29:55Z DEBUG Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", 
line 447, in start_creation


run_step(full_msg, method)

File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", 
line 437, in run_step


method()

File 
"/usr/lib/python2.7/site-packages/ipaserver/install/dsinstance.py", 
line 413, in __setup_replica


repl.setup_promote_replication(self.master_fqdn)

File 
"/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", 
line 1589, in setup_promote_replication


self.basic_replication_setup(r_conn, r_id, self.repl_man_dn, None)

File 
"/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", 
line 983, in basic_replication_setup


self.replica_config(conn, replica_id, repldn)

File 
"/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", 
line 467, in replica_config


conn.add_entry(entry)

File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 
1442, in add_entry


self.conn.add_s(str(entry.dn), list(attrs.items()))

File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__

self.gen.throw(type, value, traceback)

File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 
947, in error_handler


raise errors.DuplicateEntry()

DuplicateEntry: This entry already exists

2016-01-18T03:29:55Z DEBUG   [error] DuplicateEntry: This entry 
already exists


2016-01-18T03:29:55Z DEBUG   File 
"/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, 
in execute


return_value = self.run()

File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 
318, in run


cfgr.run()

File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", 
line 310, in run


self.execute()

File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", 
line 332, in execute


for nothing in self._executor():

File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", 
line 372, in __runner


self._handle_exception(exc_info)

File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", 
line 394, in _handle_exception


six.reraise(*exc_info)

File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", 
line 362, in __runner


step()

File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", 
line 359, in 


step = lambda: next(self.__gen)

File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", 
line 81, in run_generator_with_yield_from


six.reraise(*exc_info)

File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", 
line 59, in run_generator_with_yield_from


value = gen.send(prev_value)

File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", 
line 571, in _configure


next(executor)

File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", 
line 372, in __runner


self._handle_exception(exc_info)

File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", 
line 449, in _handle_exception


self.__parent._handle_exception(exc_info)

File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", 
line 394, in _handle_exception


six.reraise(*exc_info)

File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", 
line 446, in _handle_exception


super(ComponentBase, self)._handle_exception(exc_info)

File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", 
line 394, in _handle_exception


six.reraise(*exc_info)

File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", 
line 362, in __runner


step()

File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", 
line 359, in 


step = lambda: next(self.__gen)

File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", 
line 81, in run_generator_with_yield_from


six.reraise(*exc_info)

File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", 
line 59, in run_generator_with_yield_from


value = gen.send(prev_value)

File 

Re: [Freeipa-users] Replication failing on FreeIPA 4.2.0

[Ludwig Krispenz]

On 01/15/2016 08:32 AM, Nathan Peters wrote:

I think I've finally started to make some progress on this.  I did a lot of 
googling and found some stuff to run manually in 389 ds through ldapmodify 
commands to clean RUVs.  During this process the server crashed and when it 
came back online, suddenly all my ghost RUVs were visible through 
ipa-replica-manage list-ruv.  It was really strange, I had like 5 of them from 
winsync agreements that kept failing and needing re-initialization, and another 
5 from my earlier re-installations of the 2 other domain controllers.

I ran some more ruv cleanup commands through ldap and they all appear to be 
gone.  I'm not sure how the crash suddenly made them visible though or why they 
had to be cleaned through ldapmodify directly and ipa-replica-manage could 
neither see nor clean them.
After a crash the RUV could be rebuilt from the changelog, and the 
changelog could contain references to cleaned ReplicaIds and so they 
came to live again. The cleanallruv task was enhanced to also clean the 
changelog, but this fix is in 1.3.4.2+.

Console logs below in case anyone can shed some light on it.  I've re-installed 
the replicas again, and I'm hoping it doesn't crash in 12 hours like last time 
...

--- console output ---

[root@dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage del 
dc2-ipa-dev-nvan.mydomain.net --force --cleanup
Connection to 'dc2-ipa-dev-nvan.mydomain.net' failed: Insufficient access: 
SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context 
Invalid credentials
Forcing removal of dc2-ipa-dev-nvan.mydomain.net
Skipping calculation to determine if one or more masters would be orphaned.
Deleting replication agreements between dc2-ipa-dev-nvan.mydomain.net and 
dc1-ipa-dev-van.mydomain.net, dc1-ipa-dev-nvan.mydomain.net
Failed to get list of agreements from 'dc2-ipa-dev-nvan.mydomain.net': 
Insufficient access: SASL(-13): authentication failure: GSSAPI Failure: 
gss_accept_sec_context Invalid credentials
Forcing removal on 'dc1-ipa-dev-van.mydomain.net'
Any DNA range on 'dc2-ipa-dev-nvan.mydomain.net' will be lost
Deleted replication agreement from 'dc1-ipa-dev-van.mydomain.net' to 
'dc2-ipa-dev-nvan.mydomain.net'
Failed to determine agreement type for 'dc2-ipa-dev-nvan.mydomain.net': 
Insufficient access: SASL(-13): authentication failure: GSSAPI Failure: 
gss_accept_sec_context Invalid credentials
There were issues removing a connection for dc2-ipa-dev-nvan.mydomain.net from 
dc1-ipa-dev-nvan.mydomain.net: local variable 'type1' referenced before 
assignment
Background task created to clean replication data. This may take a while.
This may be safely interrupted with Ctrl+C
[root@dc1-ipa-dev-van slapd-mydomain-NET]#

[root@dc2-ipa-dev-nvan slapd-mydomain-NET]# ipa-server-install --uninstall

This is a NON REVERSIBLE operation and will delete all data and configuration!

Are you sure you want to continue with the uninstall procedure? [no]: yes

Replication agreements with the following IPA masters found: dc1-ipa-dev-van
.mydomain.net. Removing any replication agreements before uninstalling
the server is strongly recommended. You can remove replication agreements by
running the following command on any other IPA master:
$ ipa-replica-manage del dc2-ipa-dev-nvan.mydomain.net

Are you sure you want to continue with the uninstall procedure? [no]: yes
Shutting down all IPA services
Removing IPA client configuration
Unconfiguring ntpd
Configuring certmonger to stop tracking system certificates for KRA
Configuring certmonger to stop tracking system certificates for CA
Unconfiguring CA
Unconfiguring named
Unconfiguring ipa-dnskeysyncd
Unconfiguring web server
Unconfiguring krb5kdc
Unconfiguring kadmin
Unconfiguring directory server
ipa : ERRORInstance removal failed.
ipa : ERRORFailed to remove DS instance. You may need to remove 
instance data manually
Unconfiguring ipa_memcached
Unconfiguring ipa-otpd
[root@dc2-ipa-dev-nvan slapd-mydomain-NET]#

[root@dc1-ipa-dev-van slapd-mydomain-NET]# ipa-csreplica-manage del 
dc2-ipa-dev-nvan.mydomain.net --force -v
Directory Manager password:

Unable to connect to replica dc2-ipa-dev-nvan.mydomain.net, forcing removal
Failed to get data from 'dc2-ipa-dev-nvan.mydomain.net': cannot connect to 
'ldap://dc2-ipa-dev-nvan.mydomain.net:389':
Forcing removal on 'dc1-ipa-dev-van.mydomain.net'
There were issues removing a connection: 'NoneType' object has no attribute 
'port'


[root@dc1-ipa-dev-van slapd-mydomain-NET]# ldapsearch -D "cn=directory manager" -W -b 
"o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=---))" 
nscpentrywsi
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base 

Re: [Freeipa-users] Issues with 'A replication agreement for the host already exists', when it very much doesn't

[Ludwig Krispenz]

On 12/21/2015 05:49 PM, Alex Williams wrote:
I began installing a new ipa4 replica this morning and it all went 
wrong. The ipa-replica-install script got all the way to restarting 
ipa with systemctl at the very end, having set up replication and then 
fell over, because systemctl couldn't find the ipa service. I removed 
the replica from our master, I deleted the host from there too, I 
un-installed ipa-server on the new replica machine, I even created a 
new replica-prepare script on the master, but now the server just 
errors immediately with:


A replication agreement for this host already exists. It needs to 
be removed.


I've verified several times, that no replica, or host with the same 
name exists in the master, there are no ldap entries under masters, 
with that hostname, nothing. There is literally no trace of the new 
host, on the old master. Running `ipa-replica-manage list` shows just 
the 3 ipa servers we have already, no sign of this new host. Yet, if I 
run `ipa-replica-manage del hostname --force` on the master, it will 
in fact say that it's forcing removal, skipping checking if anything 
will be orphaned and that no RUV records were found.


I'm now lost, I really don't know where to start with fixing this.
we should first try to get a clear picture of existing agreements and 
state of replication. Could you on all servers do the following searches 
(as directory manager)


ldapsearch -LLL -o ldif-wrap=no  . -b "cn=config" 
"objectclass=nsds5replicationagreement" nsDS5ReplicaRoot nsDS5ReplicaHost
ldapsearch -LLL -o ldif-wrap=no .. -b "cn=config" 
"objectclass=nsds5replica" nsDS5ReplicaRoot nsDS5ReplicaId nsds50ruv


Not sure if this is relevant or not, but I'd rather bring it up and it 
not be, than not mention it and it turn out to be the reason. Our yum 
mirror is unfortunately now holding rhel7.2 packages, whilst our 
servers, are still on rhel7.1, which means our existing IPA servers, 
are ipa4.1 and the new one I tried to install, was ipa4.2, but on a 
rhel7.1 box. I had previously attributed the failed systemctl command, 
to the fact that I was trying to run ipa4.2 on a rhel7.1 box, as I'm 
told there were a lot of modifications to systemctl in rhel7.2, but I 
need to fix this replication agreement issue, before I can try again 
with the box upgraded to rhel7.2.


Any ideas?

Cheers

Alex



--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Purge old entries in /var/lib/dirsrv/slapd-xxx/cldb/xxx.db4 file

[Ludwig Krispenz]

Hi,

On 12/22/2015 11:43 AM, David Goudet wrote:

Hi,

I have multimaster replication environment. On each replica, folder 
/var/lib/dirsrv/slapd-/cldb/ has big size (3~GB) and old entries in 
/var/lib/dirsrv/slapd-xxx/cldb/xxx.db4 have three month year old:

sudo dbscan -f 
/var/lib/dirsrv/slapd-/cldb/ef155b03-dda611e2-a156db20-90xxx06_51c9aed900xx000.db4
 | less
dbid: 56239e5e0004
 replgen: 1445174777 Sun Oct 18 15:26:17 2015
 csn: 56239e5e0004
 uniqueid: e55d5e01-26f211e4-9b60db20-90c3b706
 dn: 
 operation: modify
 krbLastSuccessfulAuth: 20151018132617Z
 modifiersname: cn=Directory Manager
 modifytimestamp: 20151018132617Z
 entryusn: 68030946

My questions are:

a) How to purge old entries in file /var/lib/dirsrv/slapd-xxx/cldb/xxx.db4? 
(what is the procedure)
b) What is the right configuration to limit increase of this file?
setting changelog maxage should be sufficient to trim changes, but the 
age is not the only condition deciding if a recored in the changelog can 
be deleted.
- for each replicaID the last record will never be deleted, independent 
of its age, so if you have replicas in your topology which are not (or 
not frequently) updated directly there will be old changes in the changelog
- if the replica where the trimming is run and if it has replication 
agreements to other replicas, changes which were not yet replicated to 
the other replica will not be purged. So, if you have some stale 
agreements to other replicas this could prevent trimming as well.


Also trimming removes changelog records and frees space internally ro th 
edb4 file  to be reused, but it will not shrink the file size




This topic has been already talk on 
https://www.redhat.com/archives/freeipa-users/2013-February/msg00433.html or 
https://www.redhat.com/archives/freeipa-users/2015-April/msg00573.html but no 
response work for me.
Response here seems to be not applicable 
https://bugzilla.redhat.com/show_bug.cgi?id=1181341 (Centos 7, Fixed In 
Version: 389-ds-base-1.3.4.0-1.el7)

I used some attributes from the docuementation: 
https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/9.0/html/Configuration_Command_and_File_Reference/Core_Server_Configuration_Reference.html#cnchangelog5-nsslapd_changelogdir.
 Old entries are not purged and file increase even after restart service 
(service dirvsrv start and service dirvsrv stop).

(This test environment values)
dn: cn=changelog5,cn=config
objectClass: top
objectClass: extensibleobject
cn: changelog5
...
nsslapd-changelogmaxentries: 100
nsslapd-changelogmaxage: 4m

dn: cn=replica,cn=x,cn=mapping tree,cn=config
cn: replica
nsDS5Flags: 1
objectClass: top
objectClass: nsds5replica
objectClass: extensibleobject
nsDS5ReplicaType: 3
nsDS5ReplicaRoot: dc=x
nsds5ReplicaLegacyConsumer: off
nsDS5ReplicaId: 6
nsDS5ReplicaBindDN: cn=replication manager,cn=config
nsDS5ReplicaBindDN: krbprincipalname=ldap/xx
  .LYRA,cn=services,cn=accounts,dc=x
nsState:: x
nsDS5ReplicaName: d9663d08-a80f11e5-aa48d241-0b88f012
nsds5ReplicaTombstonePurgeInterval: 200
nsds5ReplicaPurgeDelay: 200
nsds5ReplicaChangeCount: 3091
nsds5replicareapactive: 0

Hereafter some informations about my environment:
CentOS release 6.5 (Final)
389-ds-base-libs-1.2.11.15-65.el6_7.x86_64
389-ds-base-1.2.11.15-65.el6_7.x86_64
ipa-client-3.0.0-47.el6.centos.1.x86_64
ipa-server-3.0.0-47.el6.centos.1.x86_64

Thanks for your help!

David



--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Restricting access to unencrypted LDAP connections

[Ludwig Krispenz]

you could set minssf:
https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/10/html/Administration_Guide/SecureConnections.html#requiring-secure-connections

On 11/18/2015 07:24 AM, Prashant Bapat wrote:

Hi,

We have a pair of freeipa servers (4.1.4) and a bunch of Linux clients 
configured to talk to them thru pam-nss-ldapd (no sssd). I want to 
ensure that these clients only talk to freeipa's LDAP server either 
via ldaps or ldap+starttls. Plain ldap should not be allowed.


I can always switch to ldaps only and close the tcp/389 port on the 
firewall. But is there a way to achieve this using tcp/389 port.?


Any suggestions appreciated.

Thanks.
--Prashant




-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] krb5kdc will not start (kerberos authentication error)

[Ludwig Krispenz]

it was a typo, try

nsslapd-accesslog-level

On 11/10/2015 03:53 PM, Gronde, Christopher (Contractor) wrote:

Ran into an error trying to set that

# ldapmodify -a -D "cn=directory manager" -W
Enter LDAP Password:
dn: cn=config
changetype: modify
replace: nsslapd-acesslog-level
: 260

modifying entry "cn=config"
ldap_modify: Server is unwilling to perform (53)
 additional info: Unknown attribute nsslapd-acesslog-level will be 
ignored

[root@comipa02 ~]# ldapmodify -a -D "cn=config" -W
Enter LDAP Password:
ldap_bind: Inappropriate authentication (48)

-----Original Message-
From: Ludwig Krispenz [mailto:lkris...@redhat.com]
Sent: Tuesday, November 10, 2015 9:48 AM
To: Gronde, Christopher (Contractor) <christopher.gro...@fincen.gov>
Cc: freeipa-users@redhat.com
Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos authentication 
error)


On 11/10/2015 03:32 PM, Gronde, Christopher (Contractor) wrote:

How do I change that log setting?  Is that done in LDAP?  Using ldapmodify?

yes,
ldapmodify ...
dn: cn=config
changetype: modify
replace: nsslapd-acesslog-level
nsslapd-acesslog-level: 260

-Original Message-
From: freeipa-users-boun...@redhat.com
[mailto:freeipa-users-boun...@redhat.com] On Behalf Of Ludwig Krispenz
Sent: Tuesday, November 10, 2015 9:03 AM
To: freeipa-users@redhat.com
Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos
authentication error)


On 11/10/2015 02:40 PM, Alexander Bokovoy wrote:

On Tue, 10 Nov 2015, Gronde, Christopher (Contractor) wrote:

Where can I verify or change the credentials it is trying to use?
Is it my LDAP password?

No, according to your logs, it is your LDAP master trying to
replicate (push changes) to your LDAP replica:

[09/Nov/2015:15:02:01 -0500] conn=2 fd=64 slot=64 connection from
 to 
[09/Nov/2015:15:02:01 -0500] conn=2 op=0 BIND dn="" method=sasl
version=3 mech=GSSAPI

err=49 could also be a result if the entry which is mapped from the principal 
is not found in the directory. A bit more info could be gained by enabling 
logging of internal searches.
Set nsslapd-acesslog-level: 260

and then look what internal searches are done during the gssapi
authentication

If that is true, it would be ldap/ Kerberos principal talking
to ldap/ Kerberos principal. If that fails, it means master
and replica KDCs have different understanding of both ldap/
and ldap/ keys which most likely means keys were rotated on
master and weren't propagated to replica.

How to solve it? One possibility is to set master's hostname as KDC
address in krb5.conf on replica, forcing LDAP server on replica to
use master's KDC. I'm absolutely not sure this will actually work but
at least it allows to see if we are indeed dealing with inconsistent
state of service principals' keys.


-Original Message-
From: Alexander Bokovoy [mailto:aboko...@redhat.com]
Sent: Tuesday, November 10, 2015 8:18 AM
To: Gronde, Christopher (Contractor) <christopher.gro...@fincen.gov>
Cc: Rob Crittenden <rcrit...@redhat.com>; freeipa-users@redhat.com
Subject: Re: [Freeipa-users] krb5kdc will not start (kerberos
authentication error)

On Tue, 10 Nov 2015, Gronde, Christopher (Contractor) wrote:

When I tried to start the service again I got no response from tail
of the log, but this is a repeating entry I see in the access log

[09/Nov/2015:15:01:04 -0500] conn=1 fd=64 slot=64 connection from
127.0.0.1 to 127.0.0.1
[09/Nov/2015:15:01:04 -0500] conn=1 op=-1 fd=64 closed - B1
[09/Nov/2015:15:02:01 -0500] conn=2 fd=64 slot=64 connection from
 to 
[09/Nov/2015:15:02:01 -0500] conn=2 op=0 BIND dn="" method=sasl
version=3 mech=GSSAPI
[09/Nov/2015:15:02:01 -0500] conn=2 op=0 RESULT err=14 tag=97
nentries=0 etime=0, SASL bind in progress
[09/Nov/2015:15:02:01 -0500] conn=2 op=1 BIND dn="" method=sasl
version=3 mech=GSSAPI
[09/Nov/2015:15:02:01 -0500] conn=2 op=1 RESULT err=14 tag=97
nentries=0 etime=0, SASL bind in progress
[09/Nov/2015:15:02:01 -0500] conn=2 op=2 BIND dn="" method=sasl
version=3 mech=GSSAPI
[09/Nov/2015:15:02:01 -0500] conn=2 op=2 RESULT err=49 tag=97
nentries=0 etime=0
[09/Nov/2015:15:02:01 -0500] conn=2 op=3 UNBIND
[09/Nov/2015:15:02:01 -0500] conn=2 op=3 fd=64 closed - U1

Does anyone know what err=14 or err=49 are?

err=14 means SASL bind in progress -- i.e. multi-round processing is
ongoing. This is normal for SASL GSSAPI.

err=49 is wrong password or username, i.e. credentials were incorrect.
It may also mean that LDAP server side was unable to process
Kerberos negotiation due to not having a current Kerberos ticket for
own service
(LDAP) and trying to request it from the Kerberos KDC but Kerberos
KDC is down.


-Original Message-
From: Rob Crittenden [mailto:rcrit...@redhat.com]
Sent: Monday, November 09, 2015 3:26 PM
To: Gronde, Christopher (Contractor)
<christopher.gro...@fincen.gov>; Alexander Bokovoy
<aboko...@redhat.com>
Cc: freeipa-users@re