On 12/06/17 18:29, Mark Reynolds wrote:
>
>
> On 06/12/2017 07:32 AM, Nick Campion via FreeIPA-users wrote:
>>
>> Thanks Mark,
>>
>> So this example is a user password change using kinit, the password
>> has been changed on freeipa02 but not then replicated to the others.
>> This happens for other records, but I don't have examples of these at
>> the moment.
>>
>> As far as I'm aware, there is no fractal replication set up.
>>
> IPA uses fractional replication, and it's possible these attributes
> are ignored/skipped.  To confirm you can run this search on freeipa02:
>
> ldapsearch -D "cn=directory manager" -W -b cn=config -xLLL
> objectclass=nsds5ReplicationAgreement
Freeipa02:

dn:
cn=meTofreeipa01.mgmt.example.com,cn=replica,cn=dc\3Dipa\2Cdc\3Dexample\2Cdc\3Dcom,cn=mapping
tree,cn=config                                                                  
                                   

cn:
meTofreeipa01.mgmt.example.com                                                  
                                                                                
                                                                

description: me to
freeipa01.mgmt.example.com                                                      
                                                                                
                                                 

nsDS5ReplicaBindMethod:
SASL/GSSAPI                                                                     
                                                                                
                                                    

nsDS5ReplicaHost:
freeipa01.mgmt.example.com                                                      
                                                                                
                                                  

nsDS5ReplicaPort:
389                                                                             
                                                                                
                                                          

nsDS5ReplicaRoot:
dc=ipa,dc=example,dc=com                                                        
                                                                                
                                               

nsDS5ReplicaTransportInfo:
LDAP                                                                            
                                                                                
                                                 

nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof
idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth
krbloginfailedcount                                                             
                            

nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn
krblastsuccessfulauth krblastfailedauth
krbloginfailedcount                                                             
                                              

nsds50ruv: {replicageneration}
57867ff5000000040000                                                            
                                                                                
                                             

nsds50ruv: {replica 4 ldap://freeipa01.mgmt.example.com:389}
57867ffe000000040000
593693b7001100040000                                                            
                                                                  

nsds50ruv: {replica 3 ldap://freeipa02.mgmt.example.com:389}
57867ffa000000030000 5930e345000200030000
nsds50ruv: {replica 5 ldap://freeipa03.mgmt.example.com:389}
59355988000000050000 59369317000300050000
nsds5ReplicaEnabled: on
nsds5ReplicaStripAttrs: modifiersName modifyTimestamp
internalModifiersName internalModifyTimestamp
nsds5replicaTimeout: 120
nsruvReplicaLastModified: {replica 4
ldap://freeipa01.mgmt.example.com:389} 00000000
nsruvReplicaLastModified: {replica 3
ldap://freeipa02.mgmt.example.com:389} 00000000
nsruvReplicaLastModified: {replica 5
ldap://freeipa03.mgmt.example.com:389} 00000000
objectClass: nsds5replicationagreement
objectClass: top
objectClass: ipaReplTopoManagedAgreement
ipaReplTopoManagedAgreementState: managed agreement - controlled by
topology plugin
nsds5replicareapactive: 0
nsds5replicaLastUpdateStart: 20170613090432Z
nsds5replicaLastUpdateEnd: 20170613090432Z
nsds5replicaChangesSentSinceStartup:: MzoxNTkvMjM5ODI0NyA1OjI0LzAg
nsds5replicaLastUpdateStatus: Error (0) Replica acquired successfully:
Incremental update succeeded
nsds5replicaUpdateInProgress: FALSE
nsds5replicaLastInitStart: 19700101000000Z
nsds5replicaLastInitEnd: 19700101000000Z

dn:
cn=meTofreeipa03.mgmt.example.com,cn=replica,cn=dc\3Dipa\2Cdc\3Dexample2Cdc\3Dcom,cn=mapping
tree,cn=config
cn: meTofreeipa03.mgmt.example.com
objectClass: nsds5replicationagreement
objectClass: top
objectClass: ipaReplTopoManagedAgreement
nsDS5ReplicaTransportInfo: LDAP
nsds5ReplicaStripAttrs: modifiersName modifyTimestamp
internalModifiersName internalModifyTimestamp
nsDS5ReplicaRoot: dc=ipa,dc=example,dc=com
nsDS5ReplicaHost: freeipa03.mgmt.example.com
nsds5replicaTimeout: 120
nsDS5ReplicaPort: 389
nsDS5ReplicaBindMethod: SASL/GSSAPI
nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof
idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth
krbloginfailedcount
description: me to freeipa03.mgmt.example.com
nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn
krblastsuccessfulauth krblastfailedauth krbloginfailedcount
ipaReplTopoManagedAgreementState: managed agreement - controlled by
topology plugin
nsds5ReplicaEnabled: on
nsds50ruv: {replicageneration} 57867ff5000000040000
nsds50ruv: {replica 5 ldap://freeipa03.mgmt.example.com:389}
59355988000000050000 5936937b000200050000
nsds50ruv: {replica 3 ldap://freeipa02.mgmt.example.com:389}
57867ffa000000030000 5930e345000200030000
nsds50ruv: {replica 4 ldap://freeipa01.mgmt.example.com:389}
57867ffe000000040000 593693b7000c00040000
nsruvReplicaLastModified: {replica 5
ldap://freeipa03.mgmt.example.com:389} 00000000
nsruvReplicaLastModified: {replica 3
ldap://freeipa02.mgmt.example.com:389} 00000000
nsruvReplicaLastModified: {replica 4
ldap://freeipa01.mgmt.example.com:389} 00000000
nsds5replicareapactive: 0
nsds5replicaLastUpdateStart: 20170613090432Z
nsds5replicaLastUpdateEnd: 20170613090432Z
nsds5replicaChangesSentSinceStartup:: MzoxMzkvMTkxMDI0NSA0OjkwNS8wIA==
nsds5replicaLastUpdateStatus: Error (0) Replica acquired successfully:
Incremental update succeeded
nsds5replicaUpdateInProgress: FALSE
nsds5replicaLastInitStart: 19700101000000Z
nsds5replicaLastInitEnd: 19700101000000Z

dn:
cn=cloneAgreement1-freeipa02.mgmt.example.com-pki-tomcat,cn=replica,cn=o\3Dipaca,cn=mapping
tree,cn=config
cn: cloneAgreement1-freeipa02.mgmt.example.com-pki-tomcat
description: cloneAgreement1-freeipa02.mgmt.example.com-pki-tomcat
nsDS5ReplicaBindDN: cn=Replication Manager
masterAgreement1-freeipa02.mgmt.example.com-pki-tomcat,ou=csusers,cn=config
nsDS5ReplicaBindMethod: Simple
nsDS5ReplicaCredentials: Redacted
nsDS5ReplicaHost: freeipa01.mgmt.example.com
nsDS5ReplicaPort: 389
nsDS5ReplicaRoot: o=ipaca
nsDS5ReplicaTransportInfo: TLS
nsds50ruv: {replicageneration} 57868040000000600000
nsds50ruv: {replica 96 ldap://freeipa01.mgmt.example.com:389}
57868041000000600000 593692b2000000600000
nsds50ruv: {replica 97 ldap://freeipa02.mgmt.example.com:389}
57868050000000610000 59355a39000400610000
nsruvReplicaLastModified: {replica 96
ldap://freeipa01.mgmt.example.com:389} 00000000
nsruvReplicaLastModified: {replica 97
ldap://freeipa02.mgmt.example.com:389} 00000000
objectClass: top
objectClass: nsds5replicationagreement
objectClass: ipaReplTopoManagedAgreement
ipaReplTopoManagedAgreementState: managed agreement - controlled by
topology plugin
nsds5replicareapactive: 0
nsds5replicaLastUpdateStart: 20170613090226Z
nsds5replicaLastUpdateEnd: 20170613090226Z
nsds5replicaChangesSentSinceStartup:
nsds5replicaLastUpdateStatus: Error (0) Replica acquired successfully:
Incremental update succeeded
nsds5replicaUpdateInProgress: FALSE
nsds5replicaLastInitStart: 19700101000000Z
nsds5replicaLastInitEnd: 19700101000000Z

>
> Then please share these entries so we can see how they are
> configured.  Perhaps do this on freeipa01 as well for comparison.
Freeipa01:

dn:
cn=freeipa01.mgmt.example.com-to-freeipa03.mgmt.example.com,cn=replica,cn=dc\3Dipa\2Cdc\3Dexamle\2Cdc\3Dcom,cn=mapping
tree,cn=config
objectClass: nsds5replicationagreement
objectClass: ipaReplTopoManagedAgreement
objectClass: top
cn: freeipa01.mgmt.example.com-to-freeipa03.mgmt.example.com
nsDS5ReplicaHost: freeipa03.mgmt.example.com
nsDS5ReplicaPort: 389
nsds5replicaTimeout: 300
nsDS5ReplicaRoot: dc=ipa,dc=example,dc=com
description: freeipa01.mgmt.example.com to freeipa03.mgmt.example.com
ipaReplTopoManagedAgreementState: managed agreement - generated by
topology plugin
nsDS5ReplicaTransportInfo: LDAP
nsDS5ReplicaBindMethod: SASL/GSSAPI
nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof
idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth
krbloginfailedcount
nsds5ReplicaStripAttrs: modifiersName modifyTimestamp
internalModifiersName internalModifyTimestamp
nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn
krblastsuccessfulauth krblastfailedauth krbloginfailedcount
nsds50ruv: {replicageneration} 57867ff5000000040000
nsds50ruv: {replica 5 ldap://freeipa03.mgmt.example.com:389}
59355988000000050000 593b4a2f000300050000
nsds50ruv: {replica 3 ldap://freeipa02.mgmt.example.com:389}
57867ffa000000030000 5937cccd000300030000
nsds50ruv: {replica 4 ldap://freeipa01.mgmt.example.com:389}
57867ffe000000040000 593b4b2f000700040000
nsruvReplicaLastModified: {replica 5
ldap://freeipa03.mgmt.example.com:389} 00000000
nsruvReplicaLastModified: {replica 3
ldap://freeipa02.mgmt.example.com:389} 00000000
nsruvReplicaLastModified: {replica 4
ldap://freeipa01.mgmt.example.com:389} 00000000
nsds5replicareapactive: 0
nsds5replicaLastUpdateStart: 20170613090421Z
nsds5replicaLastUpdateEnd: 20170613090421Z
nsds5replicaChangesSentSinceStartup:: NDoxMTM0MS8yNTkyNzgxMyA=
nsds5replicaLastUpdateStatus: Error (0) Replica acquired successfully:
Incremental update succeeded
nsds5replicaUpdateInProgress: FALSE
nsds5replicaLastInitStart: 19700101000000Z
nsds5replicaLastInitEnd: 19700101000000Z

dn:
cn=meTofreeipa02.mgmt.example.com,cn=replica,cn=dc\3Dipa\2Cdc\3Dexample\2Cdc\3Dcom,cn=mapping
tree,cn=config
cn: meTofreeipa02.mgmt.example.com
description: me to freeipa02.mgmt.example.com
nsDS5ReplicaBindMethod: SASL/GSSAPI
nsDS5ReplicaHost: freeipa02.mgmt.example.com
nsDS5ReplicaPort: 389
nsDS5ReplicaRoot: dc=ipa,dc=example,dc=com
nsDS5ReplicaTransportInfo: LDAP
nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof
idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth
krbloginfailedcount
nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn
krblastsuccessfulauth krblastfailedauth krbloginfailedcount
nsds50ruv: {replicageneration} 57867ff5000000040000
nsds50ruv: {replica 3 ldap://freeipa02.mgmt.example.com:389}
57867ffa000000030000 5937ccd3000a00030000
nsds50ruv: {replica 4 ldap://freeipa01.mgmt.example.com:389}
57867ffe000000040000 593b4b2f000700040000
nsds50ruv: {replica 5 ldap://freeipa03.mgmt.example.com:389}
59355988000000050000 593b49d8000400050000
nsds5ReplicaEnabled: on
nsds5ReplicaStripAttrs: modifiersName modifyTimestamp
internalModifiersName internalModifyTimestamp
nsds5replicaTimeout: 120
nsruvReplicaLastModified: {replica 3
ldap://freeipa02.mgmt.example.com:389} 00000000
nsruvReplicaLastModified: {replica 4
ldap://freeipa01.mgmt.example.com:389} 00000000
nsruvReplicaLastModified: {replica 5
ldap://freeipa03.mgmt.example.com:389} 00000000
objectClass: nsds5replicationagreement
objectClass: top
objectClass: ipaReplTopoManagedAgreement
ipaReplTopoManagedAgreementState: managed agreement - controlled by
topology plugin
nsds5replicareapactive: 0
nsds5replicaLastUpdateStart: 20170613090421Z
nsds5replicaLastUpdateEnd: 20170613090421Z
nsds5replicaChangesSentSinceStartup::
NDoxMDkyNy8yNTYzNjIwNiA1OjM3OC8wIDA6MTQvMCA=
nsds5replicaLastUpdateStatus: Error (0) Replica acquired successfully:
Incremental update succeeded
nsds5replicaUpdateInProgress: FALSE
nsds5replicaLastInitStart: 19700101000000Z
nsds5replicaLastInitEnd: 19700101000000Z

dn:
cn=masterAgreement1-freeipa02.mgmt.example.com-pki-tomcat,cn=replica,cn=o\3Dipaca,cn=mapping
tree,cn=config
cn: masterAgreement1-freeipa02.mgmt.example.com-pki-tomcat
description: masterAgreement1-freeipa02.mgmt.example.com-pki-tomcat
nsDS5ReplicaBindDN: cn=Replication Manager
cloneAgreement1-freeipa02.mgmt.example.com-pki-tomcat,ou=csusers,cn=config
nsDS5ReplicaBindMethod: Simple
nsDS5ReplicaCredentials: Redacted
nsDS5ReplicaHost: freeipa02.mgmt.example.com
nsDS5ReplicaPort: 389
nsDS5ReplicaRoot: o=ipaca
nsDS5ReplicaTransportInfo: TLS
nsds50ruv: {replicageneration} 57868040000000600000
nsds50ruv: {replica 97 ldap://freeipa02.mgmt.example.com:389}
57868050000000610000 59355a39000400610000
nsds50ruv: {replica 96 ldap://freeipa01.mgmt.example.com:389}
57868047000000600000 593b488f000000600000
nsruvReplicaLastModified: {replica 97
ldap://freeipa02.mgmt.example.com:389} 00000000
nsruvReplicaLastModified: {replica 96
ldap://freeipa01.mgmt.example.com:389} 00000000
objectClass: top
objectClass: nsds5replicationagreement
objectClass: ipaReplTopoManagedAgreement
ipaReplTopoManagedAgreementState: managed agreement - controlled by
topology plugin
nsds5replicareapactive: 0
nsds5replicaLastUpdateStart: 20170613090225Z
nsds5replicaLastUpdateEnd: 20170613090226Z
nsds5replicaChangesSentSinceStartup:: OTY6MzM3LzAg
nsds5replicaLastUpdateStatus: Error (0) Replica acquired successfully:
Incremental update succeeded
nsds5replicaUpdateInProgress: FALSE
nsds5replicaLastInitStart: 19700101000000Z
nsds5replicaLastInitEnd: 19700101000000Z
>>
>> Freeipa01:
>>
>> # dynamic-kepler, users, accounts, ipa.example.com
>> dn: uid=dynamic-kepler,cn=users,cn=accounts,dc=ipa,dc=example,dc=com
>> uid: dynamic-kepler
>> krbLastPwdChange: 20170608170011Z
>> krbPasswordExpiration: 20170608170011Z
>>
>> Freeipa02:
>>
>> # dynamic-kepler, users, accounts, ipa.example.com
>> dn: uid=dynamic-kepler,cn=users,cn=accounts,dc=ipa,dc=example,dc=com
>> uid: dynamic-kepler
>> krbLastPwdChange: 20170608170021Z
>> krbPasswordExpiration: 20170906170021Z
>>
>> Freeipa03:
>>
>> # dynamic-kepler, users, accounts, ipa.example.com
>> dn: uid=dynamic-kepler,cn=users,cn=accounts,dc=ipa,dc=example,dc=com
>> uid: dynamic-kepler
>> krbLastPwdChange: 20170608170011Z
>> krbPasswordExpiration: 20170608170011Z
>>
>> Errors on Freeipa02:
>>
>> [08/Jun/2017:01:46:50.635529447 +0000] replica_generate_next_csn:
>> opcsn=5938ac8b000500030000 <= basecsn=5938ac8b000500040000, adjusted
>> opcsn=5938ac8b000600030000
>> [08/Jun/2017:12:16:46.497249649 +0000] replica_generate_next_csn:
>> opcsn=5939402f000500030000 <= basecsn=5939402f000800040000, adjusted
>> opcsn=5939402f000900030000
>> [08/Jun/2017:23:38:48.197750001 +0000] replica_generate_next_csn:
>> opcsn=5939e009000100030000 <= basecsn=5939e009000f00040000, adjusted
>> opcsn=5939e009001000030000
>>
>> The other nodes have no errors from this data.
>>
>> Access logs:
>>
>> Freeipa01:
>>
>> [08/Jun/2017:01:46:50.635529447 +0000] replica_generate_next_csn:
>> opcsn=5938ac8b000500030000 <= basecsn=5938ac8b000500040000, adjusted
>> opcsn=5938ac8b000600030000
>> [08/Jun/2017:12:16:46.497249649 +0000] replica_generate_next_csn:
>> opcsn=5939402f000500030000 <= basecsn=5939402f000800040000, adjusted
>> opcsn=5939402f000900030000
>> [08/Jun/2017:23:38:48.197750001 +0000] replica_generate_next_csn:
>> opcsn=5939e009000100030000 <= basecsn=5939e009000f00040000, adjusted
>> opcsn=5939e009001000030000
>>
> This is from an error log :-)
>>
>> Freeipa02:
>>
>> Shows no logs "to" the other 2 nodes.
>>
> Well it would only show incoming connections, not outgoing. 
>>
>> Freeipa03:
>>
>> [08/Jun/2017:17:10:06.343697044 +0000] conn=9237 fd=70 slot=70
>> connection from 192.168.0.12 to 192.168.0.13
>> [08/Jun/2017:19:54:05.025713675 +0000] conn=9665 fd=70 slot=70
>> connection from 192.168.0.12 to 192.168.0.13
>>
>> Freeipa02 replication logging:
>>
>> [09/Jun/2017:11:24:58.827281135 +0000] NSMMReplicationPlugin -
>> csnplCommitALL: processing data csn 593964af000900030000
>>
>> Repeats 800 - 900 time per second with a different csn.
>>
> It looks like its replicating to other replicas, but some updates are
> skipped.  This again could be fractional replication "working".  
>
> If you look through freeipa01's access log what operation is this csn
> from:   5937cccd000f00030000 ?   Could this be one of the password
> updates that is not replicated?  This update is not sent to the other
> replicas that's why I'm asking.
I cant find that csn anywhere but the error log on freeipa02. Both
servers are logging csn's around the same time, just not this one.

Cheers
Nick

>
> Thanks,
> Mark
>>
>> On 08/06/17 15:45, Mark Reynolds wrote:
>>>
>>>
>>> On 06/07/2017 10:58 AM, Nick Campion via FreeIPA-users wrote:
>>>>
>>>> Hi all,
>>>>
>>>>  
>>>>
>>>> We have a 3 master setup that is failing to replicate changes from
>>>> a particular node to the other IPA instances. The replication
>>>> status says it's all fine, however the record hasn't been changed
>>>> on the other servers. We've seen this on user password changes,
>>>> adding hosts and services. The only thing we've found that seems to
>>>> fix this temporarily is to re-initialize from the master with the
>>>> changed record. A force-sync doesn't pick up the changed record.
>>>>
>>> What is the change you making, what attribute are you updating? 
>>> Could it be possible that its being excluded by fractional
>>> replication?  Or is it all changes?
>>>
>>> Any errors in the logs on the nodes(good and bad): 
>>> /var/log/dirsrv/slapd-INSTANCE/errors
>>>
>>> Do you see replication sessions starting between the bad node and
>>> good ones?  Are they talking?  Check the access log (
>>> /var/log/dirsrv/slapd-INSTANCE/access) on a good node and look for
>>> "connection from <BAD NODE IP address>"
>>>
>>> Next would be to enable replication logging on the bad node and
>>> reproduce the problem (then disable repl logging right away), then
>>> send us the logs to look at.  See 
>>> https://access.redhat.com/documentation/en-us/red_hat_directory_server/10/html/administration_guide/managing_replication-troubleshooting_replication_related_problems
>>>
>>> Regards,
>>> Mark
>>>
>>>> Not sure what logs would be helpful to diagnose what is happening
>>>> in this setup. 
>>>>
>>>> # ipa-replica-manage -v list `hostname`
>>>> freeipa03.mgmt.example.com: replica
>>>> last init status: None
>>>> last init ended: 1970-01-01 00:00:00+00:00
>>>> last update status: Error (0) Replica acquired successfully:
>>>> Incremental update succeeded
>>>> last update ended: 2017-06-07 14:43:53+00:00
>>>> freeipa02.mgmt.example.com: replica
>>>> last init status: None
>>>> last init ended: 1970-01-01 00:00:00+00:00
>>>> last update status: Error (0) Replica acquired successfully:
>>>> Incremental update succeeded
>>>> last update ended: 2017-06-07 14:43:53+00:00
>>>>
>>>> # ldapsearch -W -x -D "cn=directory manager" -b
>>>> "cn=users,cn=accounts,dc=ipa,dc=example,dc=com"
>>>> "nsds5ReplConflict=*" \* nsds5ReplConflict
>>>> Enter LDAP Password:
>>>> # extended LDIF
>>>> #
>>>> # LDAPv3
>>>> # base <cn=users,cn=accounts,dc=ipa,dc=example,dc=com> with scope
>>>> subtree
>>>> # filter: nsds5ReplConflict=*
>>>> # requesting: * nsds5ReplConflict
>>>> #
>>>>
>>>> # search result
>>>> search: 2
>>>> result: 0 Success
>>>>
>>>> # numResponses: 1
>>>>
>>>> Any help in what else can be checked or what logs would be helpful
>>>> would be appreciated.
>>>>
>>>> Thanks
>>>>
>>>> Nick
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
>>>> To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org
>>>
>>
>>
>>
>> _______________________________________________
>> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
>> To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org
>

_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org

Reply via email to