On 06/17/2015 01:38 PM, Alexander Frolushkin wrote:

Ok, I'll try this soon, thank you!

Also, please note, most of today dups appeared when 4 of 19 servers was very busy in IO (all our servers are VMs), because dirsrv debug was enabled to gather logs for our case about

attrlist_replace - attr_replace (nsslapd-referral, ldap://xxx-rhidm0x.unix.megafon.ru:389/o%3Dipaca) failed.


This message comes if you have duplicated ReplicaID. 'list-ruv' would show you a same url appearing several times with different RID.

To be back on the original problem (dup), I wonder if it is not related to cleanruv/corrupted RUV.

A new replica26 is created and is initialized from a master and it does not contain entry 555ac936000000140000. That allows the creation of
5580f3210000001a0000.
Then replica10 sends 555ac936000000140000 to replica26. That means that changelog/ruv of replica10 contained that update, so either cleanallruv20 did not happen on replica10 or fail to clean its CL.

Replica10 has a very old update that was not known from the master used to init replica26.
Was replica10 restarted recently.

thanks
theirry

errors

Also during collection some of dirsrv instances hangs and was restarted.

WBR,

Alexander Frolushkin

Cell +79232508764

Work +79232507764

*From:*Ludwig Krispenz [mailto:lkris...@redhat.com]
*Sent:* Wednesday, June 17, 2015 5:34 PM
*To:* Alexander Frolushkin (SIB)
*Cc:* 'thierry bordaz'; freeipa-users@redhat.com
*Subject:* Re: [Freeipa-users] replication conflicts

On 06/17/2015 12:57 PM, Alexander Frolushkin wrote:

    Unfortunately, number of duplicates grows dramatically on most
    sites. Some servers already have over 40 duplicates.

    Could you please say, may I use re-initialize on falling replica
    from the good one to fix this?

If you have a good one, this should work, "dups" are only created when a replicated ADD is received for an existing entry. But what really puzzles me is that you do not have them on all servers, something weird seems to happen, this entry seems to exist whit several replicaids, and why would replica 10 replicate this 4 hrs after the replica installation.

WBR,

Alexander Frolushkin

Cell +79232508764

Work +79232507764

*From:*Ludwig Krispenz [mailto:lkris...@redhat.com]
*Sent:* Wednesday, June 17, 2015 4:35 PM
*To:* Alexander Frolushkin (SIB)
*Cc:* 'thierry bordaz'; freeipa-users@redhat.com <mailto:freeipa-users@redhat.com>
*Subject:* Re: [Freeipa-users] replication conflicts

conn=237 is from 10.99.75.82 which replica is this ?

On 06/17/2015 12:13 PM, Alexander Frolushkin wrote:

    This is not a good news, because replica id 20 is not exist for a
    some days already. It was recreated and now have id 23

    WBR,

    Alexander Frolushkin

    Cell +79232508764

    Work +79232507764

    *From:*thierry bordaz [mailto:tbor...@redhat.com]
    *Sent:* Wednesday, June 17, 2015 4:10 PM
    *To:* Alexander Frolushkin (SIB)
    *Cc:* 'Ludwig Krispenz'; freeipa-users@redhat.com
    <mailto:freeipa-users@redhat.com>
    *Subject:* Re: [Freeipa-users] replication conflicts

    On 06/17/2015 11:56 AM, Alexander Frolushkin wrote:

        Will this be enough?

        # grep "conn=237 op=93" ./access

        [17/Jun/2015:14:39:46 +0600] conn=237 op=93 ADD dn="cn=System:
        Manage Host
        Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru"

        [17/Jun/2015:14:39:46 +0600] conn=237 op=93 RESULT err=0
        tag=105 nentries=0 etime=0 csn=555ac936000000140000

    This operation is a replicated one and the CSN is from May 19th.
    So why a replica (26) created today was initialized without that
    entry ?
    This updates was originated from replica20. Was it stopped and
    restarted recently ?




    # grep "conn=293" ./access

    [17/Jun/2015:15:33:04 +0600] conn=293 fd=75 slot=75 connection
    from 10.99.75.82 to 10.61.8.2

    [17/Jun/2015:15:33:04 +0600] conn=293 op=0 BIND dn="" method=sasl
    version=3 mech=GSSAPI

    [17/Jun/2015:15:33:04 +0600] conn=293 op=0 RESULT err=14 tag=97
    nentries=0 etime=0, SASL bind in progress

    [17/Jun/2015:15:33:04 +0600] conn=293 op=1 BIND dn="" method=sasl
    version=3 mech=GSSAPI

    [17/Jun/2015:15:33:04 +0600] conn=293 op=1 RESULT err=14 tag=97
    nentries=0 etime=0, SASL bind in progress

    [17/Jun/2015:15:33:04 +0600] conn=293 op=2 BIND dn="" method=sasl
    version=3 mech=GSSAPI

    [17/Jun/2015:15:33:04 +0600] conn=293 op=2 RESULT err=0 tag=97
    nentries=0 etime=0
    
dn="krbprincipalname=ldap/msk-rhidm-03.unix.megafon...@unix.megafon.ru,cn=services,cn=accounts,dc=unix,dc=megafon,dc=ru"
    
<mailto:krbprincipalname=ldap/msk-rhidm-03.unix.megafon...@unix.megafon.ru,cn=services,cn=accounts,dc=unix,dc=megafon,dc=ru>

    WBR,

    Alexander Frolushkin

    Cell +79232508764

    Work +79232507764

    *From:*Ludwig Krispenz [mailto:lkris...@redhat.com]
    *Sent:* Wednesday, June 17, 2015 3:53 PM
    *To:* thierry bordaz
    *Cc:* Alexander Frolushkin (SIB); freeipa-users@redhat.com
    <mailto:freeipa-users@redhat.com>
    *Subject:* Re: [Freeipa-users] replication conflicts

    On 06/17/2015 11:45 AM, thierry bordaz wrote:


        On 06/17/2015 11:22 AM, Alexander Frolushkin wrote:

            This was a usual "ipa-replica-install --setup-ca
            --setup-dns" and after that ipa-adtrust-install.

            No DEL found:

            # grep "cn=System: Manage Host
            Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru"
            ./access

            [17/Jun/2015:10:08:01 +0600] conn=2 op=89 SRCH
            base="cn=System: Manage Host
            Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru"
            scope=0 filter="(objectClass=*)" attrs="ipaPermRight
            ipaPermTargetFilter ipaPermBindRuleType ipaPermissionType
            cn objectClass memberOf member ipaPermTarget
            ipaPermDefaultAttr ipaPermLocation ipaPermIncludedAttr
            ipaPermExcludedAttr"

            [17/Jun/2015:10:08:01 +0600] conn=2 op=91 ADD
            dn="cn=System: Manage Host
            Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru"


        There is something I miss. conn=2 op=91 was a direct update on
        replica26 (not replicated) because it received its own
        CSN=5580f3210000001a0000. But it created a conflict entry, so
        at that time it existed the same entry (the one created
        20150408070720Z) . So the direct update should have been rejected.

    I think the search in op=89 did not return an entry, so it was
    added in op 91, that seems to be ok, but then 4 hrs later there is
    conn=237 adding it again.

    Alexander,

    could you get the complete 'conn=237 op=93' and also the start of
    conn 293, to show where teh connection comes from





    Would you check if the replicaID=26 is unique in the topology
    (list-ruv for example) ?





    [17/Jun/2015:14:39:46 +0600] conn=237 op=93 ADD dn="cn=System:
    Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru"

    It is also possible this entry on affected servers was previously
    duplicated and not correctly managed to delete (more recent dup
    was deleted).

    Is there any natural way to fix such issues? Maybe
    ipa-replica-manage force-sync, or ipa-replica-manage re-initialize
    on affected site servers from normal servers could help?

    WBR,

    Alexander Frolushkin

    Cell +79232508764

    Work +79232507764

    *From:*thierry bordaz [mailto:tbor...@redhat.com]
    *Sent:* Wednesday, June 17, 2015 3:15 PM
    *To:* Alexander Frolushkin (SIB)
    *Cc:* 'Ludwig Krispenz'; freeipa-users@redhat.com
    <mailto:freeipa-users@redhat.com>
    *Subject:* Re: [Freeipa-users] replication conflicts

    Hello Alexander,

    How did you initialize that new replica 26.
    Either 'cn=System: Manage Host
    Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru' was not
    part of the total init data, or a DEL of that entry happened on
    replica 26 (before a new ADD) but the DEL was not replicated to
    replica12.
    Would you check in replica26 access logs if that entry was deleted ?

    thanks
    theirry

    On 06/17/2015 11:03 AM, Alexander Frolushkin wrote:

        This is correct, thank you for understanding and for helping!

        Replica with id 26 was created today, this is our new server
        which was included in domain just a few hours ago. Looks like
        this dup came right after this new replica creation.

        WBR,

        Alexander Frolushkin

        Cell +79232508764

        Work +79232507764

        *From:*Ludwig Krispenz [mailto:lkris...@redhat.com]
        *Sent:* Wednesday, June 17, 2015 2:58 PM
        *To:* Alexander Frolushkin (SIB)
        *Cc:* freeipa-users@redhat.com <mailto:freeipa-users@redhat.com>
        *Subject:* Re: [Freeipa-users] replication conflicts

        Hi,

        you did send the data directly to me, maybe not wanting to
        share them to everyone. I'll continue discussion here, trying
        to be careful.

        The "good" entry was created in April on replica 12 "0x0c"
        createTimestamp;vucsn-5524d42b0067000c0000: 20150408070720Z

        the "nsuniqueid" entry was created today on replica 26 "0x1a"
        createTimestamp;vucsn-5580f3210000001a0000: 20150617040801Z

        if the original entry would have existed on replica26 the new
        add should have been rejected, if it was not there the
        question is why.

        Do you have any additional info on replica 26, when was it
        created, was it disconnected for some time ??

        Ludwig

        On 06/17/2015 08:13 AM, Alexander Frolushkin wrote:

            Hello.

            Another example. Today appeared on servers of different site.

            Original LDIF:

            # extended LDIF

            #

            # LDAPv3

            # base <cn=System: Manage Host
            Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru>
            with scope subtree

            # filter: (objectclass=*)

            # requesting: ALL

            #

            # System: Manage Host Keytab, permissions, pbac,
            unix.megafon.ru

            dn: cn=System: Manage Host
            Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc

            =ru

            ipaPermTargetFilter: (objectclass=ipahost)

            ipaPermRight: write

            ipaPermBindRuleType: permission

            ipaPermissionType: V2

            ipaPermissionType: MANAGED

            ipaPermissionType: SYSTEM

            cn: System: Manage Host Keytab

            objectClass: ipapermission

            objectClass: top

            objectClass: groupofnames

            objectClass: ipapermissionv2

            member: cn=Host
            Enrollment,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru

            member: cn=Host
            Administrators,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru

            ipaPermDefaultAttr: krbprincipalkey

            ipaPermDefaultAttr: krblastpwdchange

            ipaPermLocation:
            cn=computers,cn=accounts,dc=unix,dc=megafon,dc=ru

            # search result

            search: 2

            result: 0 Success

            # numResponses: 2

            # numEntries: 1

            Duplicate:

            # extended LDIF

            #

            # LDAPv3

            # base <cn=System: Manage Host
            
Keytab+nsuniqueid=708bba65-14a611e5-8a48fd19-df27ff01,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru>
            with scope subtree

            # filter: (objectclass=*)

            # requesting: ALL

            #

            # System: Manage Host Keytab +
            708bba65-14a611e5-8a48fd19-df27ff01, permissio

            ns, pbac, unix.megafon.ru

            dn: cn=System: Manage Host
            Keytab+nsuniqueid=708bba65-14a611e5-8a48fd19-df27ff

            01,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru

            ipaPermTargetFilter: (objectclass=ipahost)

            ipaPermRight: write

            ipaPermBindRuleType: permission

            ipaPermissionType: V2

            ipaPermissionType: MANAGED

            ipaPermissionType: SYSTEM

            cn: System: Manage Host Keytab

            objectClass: ipapermission

            objectClass: top

            objectClass: groupofnames

            objectClass: ipapermissionv2

            member: cn=Host
            Enrollment,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru

            member: cn=Host
            Administrators,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru

            ipaPermDefaultAttr: krbprincipalkey

            ipaPermDefaultAttr: krblastpwdchange

            ipaPermLocation:
            cn=computers,cn=accounts,dc=unix,dc=megafon,dc=ru

            # search result

            search: 2

            result: 0 Success

            # numResponses: 2

            # numEntries: 1

            No other servers in IPA domain have such duplicates.

            WBR,

            Alexander Frolushkin

            Cell +79232508764

            Work +79232507764

            *From:*freeipa-users-boun...@redhat.com
            <mailto:freeipa-users-boun...@redhat.com>
            [mailto:freeipa-users-boun...@redhat.com] *On Behalf Of
            *Ludwig Krispenz
            *Sent:* Tuesday, June 16, 2015 3:52 PM
            *To:* freeipa-users@redhat.com
            <mailto:freeipa-users@redhat.com>
            *Subject:* Re: [Freeipa-users] replication conflicts

            On 06/16/2015 11:42 AM, Alexander Frolushkin wrote:

                Hello.

                Just to remind if somebody still not familiar with our
                IPA installation J

                We currently have 18 IPA servers in domain, on 8 sites
                in different regions across the Russia.

                And now, our new problem.

                Regularly we getting a nsds5ReplConflict records on
                some of our servers, very often on servers from
                specific site. Usually it is simply a doubles and we
                can remove the renamed change to get everything back.
                But why do we have them at all?

                May be someone could explain, how we can detect the
                cause of this replication conflicts?

            if you are talking about having two "duplicate" entries,
            one: uid=xxxxx,<suffix>
            one: nsuniqueid=nnnnnnnn+uid=xxxxx,<suffix>

            these entries appear if the entry uid=xxxxx was added,
            simultaneously, on two servers. I think this can happen if
            a client tries to add an entry and if it doesn't get a
            response in some time retries on another server.
            to find out which client this is you need to check on
            which servers the entries were originally added and then
            see which client was doing it







            Sometime it is moderately harmful, because, for example
            HBAC stops working on specific server while doubles still
            present.

            Thanks in forward...

            WBR,

            Alexander Frolushkin

            Cell +79232508764

            Work +79232507764

            
------------------------------------------------------------------------


            Информация в этом сообщении предназначена исключительно
            для конкретных лиц, которым она адресована. В сообщении
            может содержаться конфиденциальная информация, которая не
            может быть раскрыта или использована кем-либо, кроме
            адресатов. Если вы не адресат этого сообщения, то
            использование, переадресация, копирование или
            распространение содержания сообщения или его части
            незаконно и запрещено. Если Вы получили это сообщение
            ошибочно, пожалуйста, незамедлительно сообщите отправителю
            об этом и удалите со всем содержимым само сообщение и
            любые возможные его копии и приложения.

            The information contained in this communication is
            intended solely for the use of the individual or entity to
            whom it is addressed and others authorized to receive it.
            It may contain confidential or legally privileged
            information. The contents may not be disclosed or used by
            anyone other than the addressee. If you are not the
            intended recipient(s), any use, disclosure, copying,
            distribution or any action taken or omitted to be taken in
            reliance on it is prohibited and may be unlawful. If you
            have received this communication in error please notify us
            immediately by responding to this email and then delete
            the e-mail and all attachments and any copies thereof.

            (c)20mf50








            
------------------------------------------------------------------------


            Информация в этом сообщении предназначена исключительно
            для конкретных лиц, которым она адресована. В сообщении
            может содержаться конфиденциальная информация, которая не
            может быть раскрыта или использована кем-либо, кроме
            адресатов. Если вы не адресат этого сообщения, то
            использование, переадресация, копирование или
            распространение содержания сообщения или его части
            незаконно и запрещено. Если Вы получили это сообщение
            ошибочно, пожалуйста, незамедлительно сообщите отправителю
            об этом и удалите со всем содержимым само сообщение и
            любые возможные его копии и приложения.

            The information contained in this communication is
            intended solely for the use of the individual or entity to
            whom it is addressed and others authorized to receive it.
            It may contain confidential or legally privileged
            information. The contents may not be disclosed or used by
            anyone other than the addressee. If you are not the
            intended recipient(s), any use, disclosure, copying,
            distribution or any action taken or omitted to be taken in
            reliance on it is prohibited and may be unlawful. If you
            have received this communication in error please notify us
            immediately by responding to this email and then delete
            the e-mail and all attachments and any copies thereof.

            (c)20mf50

        ------------------------------------------------------------------------


        Информация в этом сообщении предназначена исключительно для
        конкретных лиц, которым она адресована. В сообщении может
        содержаться конфиденциальная информация, которая не может быть
        раскрыта или использована кем-либо, кроме адресатов. Если вы
        не адресат этого сообщения, то использование, переадресация,
        копирование или распространение содержания сообщения или его
        части незаконно и запрещено. Если Вы получили это сообщение
        ошибочно, пожалуйста, незамедлительно сообщите отправителю об
        этом и удалите со всем содержимым само сообщение и любые
        возможные его копии и приложения.

        The information contained in this communication is intended
        solely for the use of the individual or entity to whom it is
        addressed and others authorized to receive it. It may contain
        confidential or legally privileged information. The contents
        may not be disclosed or used by anyone other than the
        addressee. If you are not the intended recipient(s), any use,
        disclosure, copying, distribution or any action taken or
        omitted to be taken in reliance on it is prohibited and may be
        unlawful. If you have received this communication in error
        please notify us immediately by responding to this email and
        then delete the e-mail and all attachments and any copies thereof.

        (c)20mf50






    ------------------------------------------------------------------------


    Информация в этом сообщении предназначена исключительно для
    конкретных лиц, которым она адресована. В сообщении может
    содержаться конфиденциальная информация, которая не может быть
    раскрыта или использована кем-либо, кроме адресатов. Если вы не
    адресат этого сообщения, то использование, переадресация,
    копирование или распространение содержания сообщения или его части
    незаконно и запрещено. Если Вы получили это сообщение ошибочно,
    пожалуйста, незамедлительно сообщите отправителю об этом и удалите
    со всем содержимым само сообщение и любые возможные его копии и
    приложения.

    The information contained in this communication is intended solely
    for the use of the individual or entity to whom it is addressed
    and others authorized to receive it. It may contain confidential
    or legally privileged information. The contents may not be
    disclosed or used by anyone other than the addressee. If you are
    not the intended recipient(s), any use, disclosure, copying,
    distribution or any action taken or omitted to be taken in
    reliance on it is prohibited and may be unlawful. If you have
    received this communication in error please notify us immediately
    by responding to this email and then delete the e-mail and all
    attachments and any copies thereof.

    (c)20mf50

    ------------------------------------------------------------------------


    Информация в этом сообщении предназначена исключительно для
    конкретных лиц, которым она адресована. В сообщении может
    содержаться конфиденциальная информация, которая не может быть
    раскрыта или использована кем-либо, кроме адресатов. Если вы не
    адресат этого сообщения, то использование, переадресация,
    копирование или распространение содержания сообщения или его части
    незаконно и запрещено. Если Вы получили это сообщение ошибочно,
    пожалуйста, незамедлительно сообщите отправителю об этом и удалите
    со всем содержимым само сообщение и любые возможные его копии и
    приложения.

    The information contained in this communication is intended solely
    for the use of the individual or entity to whom it is addressed
    and others authorized to receive it. It may contain confidential
    or legally privileged information. The contents may not be
    disclosed or used by anyone other than the addressee. If you are
    not the intended recipient(s), any use, disclosure, copying,
    distribution or any action taken or omitted to be taken in
    reliance on it is prohibited and may be unlawful. If you have
    received this communication in error please notify us immediately
    by responding to this email and then delete the e-mail and all
    attachments and any copies thereof.

    (c)20mf50

    ------------------------------------------------------------------------


    Информация в этом сообщении предназначена исключительно для
    конкретных лиц, которым она адресована. В сообщении может
    содержаться конфиденциальная информация, которая не может быть
    раскрыта или использована кем-либо, кроме адресатов. Если вы не
    адресат этого сообщения, то использование, переадресация,
    копирование или распространение содержания сообщения или его части
    незаконно и запрещено. Если Вы получили это сообщение ошибочно,
    пожалуйста, незамедлительно сообщите отправителю об этом и удалите
    со всем содержимым само сообщение и любые возможные его копии и
    приложения.

    The information contained in this communication is intended solely
    for the use of the individual or entity to whom it is addressed
    and others authorized to receive it. It may contain confidential
    or legally privileged information. The contents may not be
    disclosed or used by anyone other than the addressee. If you are
    not the intended recipient(s), any use, disclosure, copying,
    distribution or any action taken or omitted to be taken in
    reliance on it is prohibited and may be unlawful. If you have
    received this communication in error please notify us immediately
    by responding to this email and then delete the e-mail and all
    attachments and any copies thereof.

    (c)20mf50

------------------------------------------------------------------------


Информация в этом сообщении предназначена исключительно для конкретных лиц, которым она адресована. В сообщении может содержаться конфиденциальная информация, которая не может быть раскрыта или использована кем-либо, кроме адресатов. Если вы не адресат этого сообщения, то использование, переадресация, копирование или распространение содержания сообщения или его части незаконно и запрещено. Если Вы получили это сообщение ошибочно, пожалуйста, незамедлительно сообщите отправителю об этом и удалите со всем содержимым само сообщение и любые возможные его копии и приложения.

The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. The contents may not be disclosed or used by anyone other than the addressee. If you are not the intended recipient(s), any use, disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. If you have received this communication in error please notify us immediately by responding to this email and then delete the e-mail and all attachments and any copies thereof.

(c)20mf50


------------------------------------------------------------------------

Информация в этом сообщении предназначена исключительно для конкретных лиц, которым она адресована. В сообщении может содержаться конфиденциальная информация, которая не может быть раскрыта или использована кем-либо, кроме адресатов. Если вы не адресат этого сообщения, то использование, переадресация, копирование или распространение содержания сообщения или его части незаконно и запрещено. Если Вы получили это сообщение ошибочно, пожалуйста, незамедлительно сообщите отправителю об этом и удалите со всем содержимым само сообщение и любые возможные его копии и приложения.

The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. The contents may not be disclosed or used by anyone other than the addressee. If you are not the intended recipient(s), any use, disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. If you have received this communication in error please notify us immediately by responding to this email and then delete the e-mail and all attachments and any copies thereof.

(c)20mf50

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Reply via email to