Re: [Freeipa-users] be_pam_handler_callback Backend returned: (3, 4, ) [Internal Error (System error)]

[Ludwig Krispenz]

On 01/18/2017 08:13 AM, Harald Dunkel wrote:
Hi Ludwig,
On 01/17/17 17:01, Ludwig Krispenz wrote:
On 01/17/2017 04:48 PM, Harald Dunkel wrote:
On 01/17/17 16:12, Harald Dunkel wrote:
On 01/17/17 11:38, Sumit Bose wrote:
On Tue, Jan 17, 2017 at 10:44:14AM +0100, Harald Dunkel wrote:
It seems something got corrupted in my ipa setup. I found this in the
sssd log file on Wheezy:

(Tue Jan 17 10:19:02 2017) [hbac_shost_attrs_to_rule] (0x0400): Processing 
source hosts for rule [allow_all]
(Tue Jan 17 10:19:02 2017) [hbac_eval_user_element] (0x0080): Parse error on 
[cn=System: Manage Host 
Principals+nsuniqueid=109be36e-ccd911e6-a5b3d0c8-d8da17db,cn=permissions,cn=pbac,dc=example,dc=de]
Looks like there was a replication conflict, please see
https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/10/html/Administration_Guide/Managing_Replication-Solving_Common_Replication_Conflicts.html
how to resolve it.

% ldapsearch -D "cn=directory manager" -w secret -b "dc=example,dc=de" 
"nsds5ReplConflict=*" \* nsds5ReplConflict | grep nsds5ReplConflict | wc -l
26

PS:

nsds5ReplConflict: namingConflict 
cn=ipaservers,cn=hostgroups,cn=accounts,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=ipaservers,cn=ng,cn=alt,dc=example,dc=de
nsds5ReplConflict: namingConflict 
cn=domain,cn=topology,cn=ipa,cn=etc,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=locations,cn=etc,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=dns 
administrators,cn=privileges,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=dns 
servers,cn=privileges,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=cas,cn=ca,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=custodia,cn=ipa,cn=etc,dc=example,dc=de
nsds5ReplConflict: namingConflict 
cn=dogtag,cn=custodia,cn=ipa,cn=etc,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=system: add 
ca,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=system: delete 
ca,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=system: modify 
ca,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=system: read 
cas,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=system: modify dns servers 
configuration,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=system: read dns servers 
configuration,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=System: Manage Host 
Principals,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=System: Add IPA 
Locations,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=System: Modify IPA 
Locations,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=System: Read IPA 
Locations,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=System: Remove IPA 
Locations,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=System: Read Locations of IPA 
Servers,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=System: Read Status of Services on IPA 
Servers,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=System: Manage Service 
Principals,cn=permissions,cn=pbac,dc=example,dc=de
nsds5ReplConflict: namingConflict cn=System: Manage User 
Principals,cn=permissions,cn=pbac,dc=example,dc=de

This looks like a problem of ipa-server-install. These entries were created
in the very first seconds.
Conflict entries are created if an entry is added on different servers at the "same 
time", where same time means it is created on instance x before the add of the entry 
on instance y was replicated to x. This can happen if you run things in parallel, eg 
upgrades.

You mean Freeipa has a race condition? I use tools like clusterssh to
install or upgrade several hosts in parallel (n <= 49 due to available
screen and font size). The "same time" is built in.

Of course I understand that Freeipa is a special case, because it is
network application, but it should be able to handle n = 2.
it is handling it (389-ds does), but the way it is handling it has the 
conflict entries as a result.

If you add an entry to an instance it is assigned a unique identifier, 
the nsuniqeuid. The dn of an entry can change by modrdn, but the 
nsuniqueid is not changed. The nsuniqueid is also used in replication to 
identify which is the entry the replicated op has to be applied to.If 
you add entries with the same dn in parallel you have entries with the 
same dn but different nsuniqueids, this cannot be prevented, on instance 
has no control of what happens on another instance.
But if these entries are replicated you have two entries with the same 
dn but different nsuniqueids on the same server. This violates the ldap 
data model and ds needs to handle it. The way we do it is to transform 
one of the entries (always the same on all instances) into a conflict entry.

There have been several proposal to change this and there is work in 
progress to  make the conflict enties invisible (only visible on 
explicite request) to client operations. But this is no trivial task, 
the main problem is that you can not only add entries with the same dn 
in parallel, but you can also add children to these entries, or rename 
them, or delete them before they are replicated and then, if you are 
hiding them entries will have no visible parent or there will be 
problems in deletion.
I definitely want to resolve this, but all proposed and tried solutions 
ended up in some unresolved issues, corner case but still existent.


There is no simple way to get rid of them, you need to delete them one by one, 
so do:
ldapmodify .......
dn:  cn=System: Manage Host 
Principals+nsuniqueid=109be36e-ccd911e6-a5b3d0c8-d8da17db,cn=permissions,cn=pbac,dc=example,dc=de
changetype: delete

for all of your conflict entries
I am surely no specialist for ldap, so hopefully its allowed to ask
a question:

This is a tree-like structure. If I delete a conflicting node, what
happens to the leafs? Is there any indication that these leafs
contain information that is not needed anymore? Isn't it possible
that server b created a huge tree with tons of subnodes and leafs
before the conflict is detected?
if you try to delete a node which has children the deletion will be 
rejected. The bad thing is that you could have conflict entries with 
children. In that case you needd to decide if you want to delete them or 
move them to the "valid" entries before deleting the conflict entry



Every helpful comment is highly appreciated
Harri


--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project