Re: [Freeipa-users] replication conflicts
Hello!
Thanks, currently I'm trying to re-initialize all our replicas, hope this will
fix most issues.
WBR,
Alexander Frolushkin
Cell +79232508764
Work +79232507764
From: thierry bordaz [mailto:tbor...@redhat.com]
Sent: Wednesday, June 17, 2015 6:40 PM
To: Alexander Frolushkin (SIB)
Cc: 'Ludwig Krispenz'; freeipa-users@redhat.com
Subject: Re: [Freeipa-users] replication conflicts
On 06/17/2015 02:27 PM, Alexander Frolushkin wrote:
Except:
unable to decode: {replica 22} 5576b83e00020016 5576ba4b00020016
unable to decode: {replica 20} 55716e5700030014 55716e5700030014
unable to decode: {replica 16} 548a81260010 548a81260010
unable to decode: {replica 24} 557fb7d400040018 557fb9a100100018
unable to decode: {replica 21} 5576ac960015 5576b52e00020015
records, all replicas are unique and have corresponding id's.
Total number of not unable to decode servers is equal to real number of
replicas in domain.
I can confirm some of cleanallruv processes was not finished correctly after
some replica remove.
Slapd on replica id 10 last restarted yesterday 15:05 server local time.
It was restarted 16/Jun/2015:15:05 and send the update one day after !
[17/Jun/2015:14:39:46 +0600] conn=237 op=93 ADD dn=cn=System: Manage Host
Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru
[17/Jun/2015:14:39:46 +0600] conn=237 op=93 RESULT err=0 tag=105 nentries=0
etime=0 csn=555ac9360014
May be it was very very slow.
The same question, may it help if I tomorrow will do re-initialize all replicas
from our relatively good-conditioned site?
If you reinit all replicas, it will clear their CL so such very old updates
will no long exist.
Now if it exists very slow replica (like replica10) that holds their updates
for long before replicating then it increases the possibility of creating
conflicts.
WBR,
Alexander Frolushkin
Cell +79232508764
Work +79232507764
From: thierry bordaz [mailto:tbor...@redhat.com]
Sent: Wednesday, June 17, 2015 6:16 PM
To: Alexander Frolushkin (SIB)
Cc: 'Ludwig Krispenz'; freeipa-users@redhat.commailto:freeipa-users@redhat.com
Subject: Re: [Freeipa-users] replication conflicts
On 06/17/2015 01:38 PM, Alexander Frolushkin wrote:
Ok, I'll try this soon, thank you!
Also, please note, most of today dups appeared when 4 of 19 servers was very
busy in IO (all our servers are VMs), because dirsrv debug was enabled to
gather logs for our case about
attrlist_replace - attr_replace (nsslapd-referral,
ldap://xxx-rhidm0x.unix.megafon.ru:389/o%3Dipaca) failed.
This message comes if you have duplicated ReplicaID. 'list-ruv' would show you
a same url appearing several times with different RID.
To be back on the original problem (dup), I wonder if it is not related to
cleanruv/corrupted RUV.
A new replica26 is created and is initialized from a master and it does not
contain entry 555ac9360014. That allows the creation of
5580f321001a.
Then replica10 sends 555ac9360014 to replica26. That means that
changelog/ruv of replica10 contained that update, so either cleanallruv20 did
not happen on replica10 or fail to clean its CL.
Replica10 has a very old update that was not known from the master used to init
replica26.
Was replica10 restarted recently.
thanks
theirry
errors
Also during collection some of dirsrv instances hangs and was restarted.
WBR,
Alexander Frolushkin
Cell +79232508764
Work +79232507764
From: Ludwig Krispenz [mailto:lkris...@redhat.com]
Sent: Wednesday, June 17, 2015 5:34 PM
To: Alexander Frolushkin (SIB)
Cc: 'thierry bordaz'; freeipa-users@redhat.commailto:freeipa-users@redhat.com
Subject: Re: [Freeipa-users] replication conflicts
On 06/17/2015 12:57 PM, Alexander Frolushkin wrote:
Unfortunately, number of duplicates grows dramatically on most sites. Some
servers already have over 40 duplicates.
Could you please say, may I use re-initialize on falling replica from the good
one to fix this?
If you have a good one, this should work, dups are only created when a
replicated ADD is received for an existing entry.
But what really puzzles me is that you do not have them on all servers,
something weird seems to happen, this entry seems to exist whit several
replicaids, and why would replica 10 replicate this 4 hrs after the replica
installation.
WBR,
Alexander Frolushkin
Cell +79232508764
Work +79232507764
From: Ludwig Krispenz [mailto:lkris...@redhat.com]
Sent: Wednesday, June 17, 2015 4:35 PM
To: Alexander Frolushkin (SIB)
Cc: 'thierry bordaz'; freeipa-users@redhat.commailto:freeipa-users@redhat.com
Subject: Re: [Freeipa-users] replication conflicts
conn=237 is from 10.99.75.82 which replica is this ?
On 06/17/2015 12:13 PM, Alexander Frolushkin wrote:
This is not a good news, because replica id 20 is not exist for a some days
already. It was recreated and now have id 23
WBR,
Alexander Frolushkin
Cell +79232508764
Work +79232507764
From: thierry bordaz [mailto:tbor...@redhat.com
Re: [Freeipa-users] replication conflicts
On 06/17/2015 01:38 PM, Alexander Frolushkin wrote: Ok, I'll try this soon, thank you! Also, please note, most of today dups appeared when 4 of 19 servers was very busy in IO (all our servers are VMs), because dirsrv debug was enabled to gather logs for our case about attrlist_replace - attr_replace (nsslapd-referral, ldap://xxx-rhidm0x.unix.megafon.ru:389/o%3Dipaca) failed. This message comes if you have duplicated ReplicaID. 'list-ruv' would show you a same url appearing several times with different RID. To be back on the original problem (dup), I wonder if it is not related to cleanruv/corrupted RUV. A new replica26 is created and is initialized from a master and it does not contain entry 555ac9360014. That allows the creation of 5580f321001a. Then replica10 sends 555ac9360014 to replica26. That means that changelog/ruv of replica10 contained that update, so either cleanallruv20 did not happen on replica10 or fail to clean its CL. Replica10 has a very old update that was not known from the master used to init replica26. Was replica10 restarted recently. thanks theirry errors Also during collection some of dirsrv instances hangs and was restarted. WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 *From:*Ludwig Krispenz [mailto:lkris...@redhat.com] *Sent:* Wednesday, June 17, 2015 5:34 PM *To:* Alexander Frolushkin (SIB) *Cc:* 'thierry bordaz'; freeipa-users@redhat.com *Subject:* Re: [Freeipa-users] replication conflicts On 06/17/2015 12:57 PM, Alexander Frolushkin wrote: Unfortunately, number of duplicates grows dramatically on most sites. Some servers already have over 40 duplicates. Could you please say, may I use re-initialize on falling replica from the good one to fix this? If you have a good one, this should work, dups are only created when a replicated ADD is received for an existing entry. But what really puzzles me is that you do not have them on all servers, something weird seems to happen, this entry seems to exist whit several replicaids, and why would replica 10 replicate this 4 hrs after the replica installation. WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 *From:*Ludwig Krispenz [mailto:lkris...@redhat.com] *Sent:* Wednesday, June 17, 2015 4:35 PM *To:* Alexander Frolushkin (SIB) *Cc:* 'thierry bordaz'; freeipa-users@redhat.com mailto:freeipa-users@redhat.com *Subject:* Re: [Freeipa-users] replication conflicts conn=237 is from 10.99.75.82 which replica is this ? On 06/17/2015 12:13 PM, Alexander Frolushkin wrote: This is not a good news, because replica id 20 is not exist for a some days already. It was recreated and now have id 23 WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 *From:*thierry bordaz [mailto:tbor...@redhat.com] *Sent:* Wednesday, June 17, 2015 4:10 PM *To:* Alexander Frolushkin (SIB) *Cc:* 'Ludwig Krispenz'; freeipa-users@redhat.com mailto:freeipa-users@redhat.com *Subject:* Re: [Freeipa-users] replication conflicts On 06/17/2015 11:56 AM, Alexander Frolushkin wrote: Will this be enough? # grep conn=237 op=93 ./access [17/Jun/2015:14:39:46 +0600] conn=237 op=93 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru [17/Jun/2015:14:39:46 +0600] conn=237 op=93 RESULT err=0 tag=105 nentries=0 etime=0 csn=555ac9360014 This operation is a replicated one and the CSN is from May 19th. So why a replica (26) created today was initialized without that entry ? This updates was originated from replica20. Was it stopped and restarted recently ? # grep conn=293 ./access [17/Jun/2015:15:33:04 +0600] conn=293 fd=75 slot=75 connection from 10.99.75.82 to 10.61.8.2 [17/Jun/2015:15:33:04 +0600] conn=293 op=0 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:15:33:04 +0600] conn=293 op=0 RESULT err=14 tag=97 nentries=0 etime=0, SASL bind in progress [17/Jun/2015:15:33:04 +0600] conn=293 op=1 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:15:33:04 +0600] conn=293 op=1 RESULT err=14 tag=97 nentries=0 etime=0, SASL bind in progress [17/Jun/2015:15:33:04 +0600] conn=293 op=2 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:15:33:04 +0600] conn=293 op=2 RESULT err=0 tag=97 nentries=0 etime=0 dn=krbprincipalname=ldap/msk-rhidm-03.unix.megafon...@unix.megafon.ru,cn=services,cn=accounts,dc=unix,dc=megafon,dc=ru mailto:krbprincipalname=ldap/msk-rhidm-03.unix.megafon...@unix.megafon.ru,cn=services,cn=accounts,dc=unix,dc=megafon,dc=ru WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 *From:*Ludwig Krispenz [mailto:lkris...@redhat.com] *Sent:* Wednesday, June 17, 2015 3:53 PM *To:* thierry bordaz *Cc:* Alexander Frolushkin (SIB); freeipa-users
Re: [Freeipa-users] replication conflicts
Except:
unable to decode: {replica 22} 5576b83e00020016 5576ba4b00020016
unable to decode: {replica 20} 55716e5700030014 55716e5700030014
unable to decode: {replica 16} 548a81260010 548a81260010
unable to decode: {replica 24} 557fb7d400040018 557fb9a100100018
unable to decode: {replica 21} 5576ac960015 5576b52e00020015
records, all replicas are unique and have corresponding id's.
Total number of not unable to decode servers is equal to real number of
replicas in domain.
I can confirm some of cleanallruv processes was not finished correctly after
some replica remove.
Slapd on replica id 10 last restarted yesterday 15:05 server local time.
The same question, may it help if I tomorrow will do re-initialize all replicas
from our relatively good-conditioned site?
WBR,
Alexander Frolushkin
Cell +79232508764
Work +79232507764
From: thierry bordaz [mailto:tbor...@redhat.com]
Sent: Wednesday, June 17, 2015 6:16 PM
To: Alexander Frolushkin (SIB)
Cc: 'Ludwig Krispenz'; freeipa-users@redhat.com
Subject: Re: [Freeipa-users] replication conflicts
On 06/17/2015 01:38 PM, Alexander Frolushkin wrote:
Ok, I'll try this soon, thank you!
Also, please note, most of today dups appeared when 4 of 19 servers was very
busy in IO (all our servers are VMs), because dirsrv debug was enabled to
gather logs for our case about
attrlist_replace - attr_replace (nsslapd-referral,
ldap://xxx-rhidm0x.unix.megafon.ru:389/o%3Dipaca) failed.
This message comes if you have duplicated ReplicaID. 'list-ruv' would show you
a same url appearing several times with different RID.
To be back on the original problem (dup), I wonder if it is not related to
cleanruv/corrupted RUV.
A new replica26 is created and is initialized from a master and it does not
contain entry 555ac9360014. That allows the creation of
5580f321001a.
Then replica10 sends 555ac9360014 to replica26. That means that
changelog/ruv of replica10 contained that update, so either cleanallruv20 did
not happen on replica10 or fail to clean its CL.
Replica10 has a very old update that was not known from the master used to init
replica26.
Was replica10 restarted recently.
thanks
theirry
errors
Also during collection some of dirsrv instances hangs and was restarted.
WBR,
Alexander Frolushkin
Cell +79232508764
Work +79232507764
From: Ludwig Krispenz [mailto:lkris...@redhat.com]
Sent: Wednesday, June 17, 2015 5:34 PM
To: Alexander Frolushkin (SIB)
Cc: 'thierry bordaz'; freeipa-users@redhat.commailto:freeipa-users@redhat.com
Subject: Re: [Freeipa-users] replication conflicts
On 06/17/2015 12:57 PM, Alexander Frolushkin wrote:
Unfortunately, number of duplicates grows dramatically on most sites. Some
servers already have over 40 duplicates.
Could you please say, may I use re-initialize on falling replica from the good
one to fix this?
If you have a good one, this should work, dups are only created when a
replicated ADD is received for an existing entry.
But what really puzzles me is that you do not have them on all servers,
something weird seems to happen, this entry seems to exist whit several
replicaids, and why would replica 10 replicate this 4 hrs after the replica
installation.
WBR,
Alexander Frolushkin
Cell +79232508764
Work +79232507764
From: Ludwig Krispenz [mailto:lkris...@redhat.com]
Sent: Wednesday, June 17, 2015 4:35 PM
To: Alexander Frolushkin (SIB)
Cc: 'thierry bordaz'; freeipa-users@redhat.commailto:freeipa-users@redhat.com
Subject: Re: [Freeipa-users] replication conflicts
conn=237 is from 10.99.75.82 which replica is this ?
On 06/17/2015 12:13 PM, Alexander Frolushkin wrote:
This is not a good news, because replica id 20 is not exist for a some days
already. It was recreated and now have id 23
WBR,
Alexander Frolushkin
Cell +79232508764
Work +79232507764
From: thierry bordaz [mailto:tbor...@redhat.com]
Sent: Wednesday, June 17, 2015 4:10 PM
To: Alexander Frolushkin (SIB)
Cc: 'Ludwig Krispenz'; freeipa-users@redhat.commailto:freeipa-users@redhat.com
Subject: Re: [Freeipa-users] replication conflicts
On 06/17/2015 11:56 AM, Alexander Frolushkin wrote:
Will this be enough?
# grep conn=237 op=93 ./access
[17/Jun/2015:14:39:46 +0600] conn=237 op=93 ADD dn=cn=System: Manage Host
Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru
[17/Jun/2015:14:39:46 +0600] conn=237 op=93 RESULT err=0 tag=105 nentries=0
etime=0 csn=555ac9360014
This operation is a replicated one and the CSN is from May 19th. So why a
replica (26) created today was initialized without that entry ?
This updates was originated from replica20. Was it stopped and restarted
recently ?
# grep conn=293 ./access
[17/Jun/2015:15:33:04 +0600] conn=293 fd=75 slot=75 connection from 10.99.75.82
to 10.61.8.2
[17/Jun/2015:15:33:04 +0600] conn=293 op=0 BIND dn= method=sasl version=3
mech=GSSAPI
[17/Jun/2015:15:33:04 +0600] conn=293 op=0 RESULT err=14 tag=97 nentries=0
etime=0, SASL bind
Re: [Freeipa-users] replication conflicts
Ok, I'll try this soon, thank you! Also, please note, most of today dups appeared when 4 of 19 servers was very busy in IO (all our servers are VMs), because dirsrv debug was enabled to gather logs for our case about attrlist_replace - attr_replace (nsslapd-referral, ldap://xxx-rhidm0x.unix.megafon.ru:389/o%3Dipaca) failed. errors Also during collection some of dirsrv instances hangs and was restarted. WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: Ludwig Krispenz [mailto:lkris...@redhat.com] Sent: Wednesday, June 17, 2015 5:34 PM To: Alexander Frolushkin (SIB) Cc: 'thierry bordaz'; freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts On 06/17/2015 12:57 PM, Alexander Frolushkin wrote: Unfortunately, number of duplicates grows dramatically on most sites. Some servers already have over 40 duplicates. Could you please say, may I use re-initialize on falling replica from the good one to fix this? If you have a good one, this should work, dups are only created when a replicated ADD is received for an existing entry. But what really puzzles me is that you do not have them on all servers, something weird seems to happen, this entry seems to exist whit several replicaids, and why would replica 10 replicate this 4 hrs after the replica installation. WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: Ludwig Krispenz [mailto:lkris...@redhat.com] Sent: Wednesday, June 17, 2015 4:35 PM To: Alexander Frolushkin (SIB) Cc: 'thierry bordaz'; freeipa-users@redhat.commailto:freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts conn=237 is from 10.99.75.82 which replica is this ? On 06/17/2015 12:13 PM, Alexander Frolushkin wrote: This is not a good news, because replica id 20 is not exist for a some days already. It was recreated and now have id 23 WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: thierry bordaz [mailto:tbor...@redhat.com] Sent: Wednesday, June 17, 2015 4:10 PM To: Alexander Frolushkin (SIB) Cc: 'Ludwig Krispenz'; freeipa-users@redhat.commailto:freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts On 06/17/2015 11:56 AM, Alexander Frolushkin wrote: Will this be enough? # grep conn=237 op=93 ./access [17/Jun/2015:14:39:46 +0600] conn=237 op=93 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru [17/Jun/2015:14:39:46 +0600] conn=237 op=93 RESULT err=0 tag=105 nentries=0 etime=0 csn=555ac9360014 This operation is a replicated one and the CSN is from May 19th. So why a replica (26) created today was initialized without that entry ? This updates was originated from replica20. Was it stopped and restarted recently ? # grep conn=293 ./access [17/Jun/2015:15:33:04 +0600] conn=293 fd=75 slot=75 connection from 10.99.75.82 to 10.61.8.2 [17/Jun/2015:15:33:04 +0600] conn=293 op=0 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:15:33:04 +0600] conn=293 op=0 RESULT err=14 tag=97 nentries=0 etime=0, SASL bind in progress [17/Jun/2015:15:33:04 +0600] conn=293 op=1 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:15:33:04 +0600] conn=293 op=1 RESULT err=14 tag=97 nentries=0 etime=0, SASL bind in progress [17/Jun/2015:15:33:04 +0600] conn=293 op=2 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:15:33:04 +0600] conn=293 op=2 RESULT err=0 tag=97 nentries=0 etime=0 dn=krbprincipalname=ldap/msk-rhidm-03.unix.megafon...@unix.megafon.ru,cn=services,cn=accounts,dc=unix,dc=megafon,dc=rumailto:krbprincipalname=ldap/msk-rhidm-03.unix.megafon...@unix.megafon.ru,cn=services,cn=accounts,dc=unix,dc=megafon,dc=ru WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: Ludwig Krispenz [mailto:lkris...@redhat.com] Sent: Wednesday, June 17, 2015 3:53 PM To: thierry bordaz Cc: Alexander Frolushkin (SIB); freeipa-users@redhat.commailto:freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts On 06/17/2015 11:45 AM, thierry bordaz wrote: On 06/17/2015 11:22 AM, Alexander Frolushkin wrote: This was a usual ipa-replica-install --setup-ca --setup-dns and after that ipa-adtrust-install. No DEL found: # grep cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru ./access [17/Jun/2015:10:08:01 +0600] conn=2 op=89 SRCH base=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru scope=0 filter=(objectClass=*) attrs=ipaPermRight ipaPermTargetFilter ipaPermBindRuleType ipaPermissionType cn objectClass memberOf member ipaPermTarget ipaPermDefaultAttr ipaPermLocation ipaPermIncludedAttr ipaPermExcludedAttr [17/Jun/2015:10:08:01 +0600] conn=2 op=91 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru There is something I miss. conn=2 op=91 was a direct update on replica26 (not replicated) because it received its own CSN=5580f321001a. But it created a conflict entry, so at that time it existed the same
Re: [Freeipa-users] replication conflicts
On 06/17/2015 12:57 PM, Alexander Frolushkin wrote: Unfortunately, number of duplicates grows dramatically on most sites. Some servers already have over 40 duplicates. Could you please say, may I use re-initialize on falling replica from the good one to fix this? If you have a good one, this should work, dups are only created when a replicated ADD is received for an existing entry. But what really puzzles me is that you do not have them on all servers, something weird seems to happen, this entry seems to exist whit several replicaids, and why would replica 10 replicate this 4 hrs after the replica installation. WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 *From:*Ludwig Krispenz [mailto:lkris...@redhat.com] *Sent:* Wednesday, June 17, 2015 4:35 PM *To:* Alexander Frolushkin (SIB) *Cc:* 'thierry bordaz'; freeipa-users@redhat.com *Subject:* Re: [Freeipa-users] replication conflicts conn=237 is from 10.99.75.82 which replica is this ? On 06/17/2015 12:13 PM, Alexander Frolushkin wrote: This is not a good news, because replica id 20 is not exist for a some days already. It was recreated and now have id 23 WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 *From:*thierry bordaz [mailto:tbor...@redhat.com] *Sent:* Wednesday, June 17, 2015 4:10 PM *To:* Alexander Frolushkin (SIB) *Cc:* 'Ludwig Krispenz'; freeipa-users@redhat.com mailto:freeipa-users@redhat.com *Subject:* Re: [Freeipa-users] replication conflicts On 06/17/2015 11:56 AM, Alexander Frolushkin wrote: Will this be enough? # grep conn=237 op=93 ./access [17/Jun/2015:14:39:46 +0600] conn=237 op=93 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru [17/Jun/2015:14:39:46 +0600] conn=237 op=93 RESULT err=0 tag=105 nentries=0 etime=0 csn=555ac9360014 This operation is a replicated one and the CSN is from May 19th. So why a replica (26) created today was initialized without that entry ? This updates was originated from replica20. Was it stopped and restarted recently ? # grep conn=293 ./access [17/Jun/2015:15:33:04 +0600] conn=293 fd=75 slot=75 connection from 10.99.75.82 to 10.61.8.2 [17/Jun/2015:15:33:04 +0600] conn=293 op=0 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:15:33:04 +0600] conn=293 op=0 RESULT err=14 tag=97 nentries=0 etime=0, SASL bind in progress [17/Jun/2015:15:33:04 +0600] conn=293 op=1 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:15:33:04 +0600] conn=293 op=1 RESULT err=14 tag=97 nentries=0 etime=0, SASL bind in progress [17/Jun/2015:15:33:04 +0600] conn=293 op=2 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:15:33:04 +0600] conn=293 op=2 RESULT err=0 tag=97 nentries=0 etime=0 dn=krbprincipalname=ldap/msk-rhidm-03.unix.megafon...@unix.megafon.ru,cn=services,cn=accounts,dc=unix,dc=megafon,dc=ru mailto:krbprincipalname=ldap/msk-rhidm-03.unix.megafon...@unix.megafon.ru,cn=services,cn=accounts,dc=unix,dc=megafon,dc=ru WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 *From:*Ludwig Krispenz [mailto:lkris...@redhat.com] *Sent:* Wednesday, June 17, 2015 3:53 PM *To:* thierry bordaz *Cc:* Alexander Frolushkin (SIB); freeipa-users@redhat.com mailto:freeipa-users@redhat.com *Subject:* Re: [Freeipa-users] replication conflicts On 06/17/2015 11:45 AM, thierry bordaz wrote: On 06/17/2015 11:22 AM, Alexander Frolushkin wrote: This was a usual ipa-replica-install --setup-ca --setup-dns and after that ipa-adtrust-install. No DEL found: # grep cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru ./access [17/Jun/2015:10:08:01 +0600] conn=2 op=89 SRCH base=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru scope=0 filter=(objectClass=*) attrs=ipaPermRight ipaPermTargetFilter ipaPermBindRuleType ipaPermissionType cn objectClass memberOf member ipaPermTarget ipaPermDefaultAttr ipaPermLocation ipaPermIncludedAttr ipaPermExcludedAttr [17/Jun/2015:10:08:01 +0600] conn=2 op=91 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru There is something I miss. conn=2 op=91 was a direct update on replica26 (not replicated) because it received its own CSN=5580f321001a. But it created a conflict entry, so at that time it existed the same entry (the one created 20150408070720Z) . So the direct update should have been rejected. I think the search in op=89 did not return an entry, so it was added in op 91, that seems
Re: [Freeipa-users] replication conflicts
Hello. Another example. Today appeared on servers of different site. Original LDIF: # extended LDIF # # LDAPv3 # base cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru with scope subtree # filter: (objectclass=*) # requesting: ALL # # System: Manage Host Keytab, permissions, pbac, unix.megafon.ru dn: cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc =ru ipaPermTargetFilter: (objectclass=ipahost) ipaPermRight: write ipaPermBindRuleType: permission ipaPermissionType: V2 ipaPermissionType: MANAGED ipaPermissionType: SYSTEM cn: System: Manage Host Keytab objectClass: ipapermission objectClass: top objectClass: groupofnames objectClass: ipapermissionv2 member: cn=Host Enrollment,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru member: cn=Host Administrators,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru ipaPermDefaultAttr: krbprincipalkey ipaPermDefaultAttr: krblastpwdchange ipaPermLocation: cn=computers,cn=accounts,dc=unix,dc=megafon,dc=ru # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 Duplicate: # extended LDIF # # LDAPv3 # base cn=System: Manage Host Keytab+nsuniqueid=708bba65-14a611e5-8a48fd19-df27ff01,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru with scope subtree # filter: (objectclass=*) # requesting: ALL # # System: Manage Host Keytab + 708bba65-14a611e5-8a48fd19-df27ff01, permissio ns, pbac, unix.megafon.ru dn: cn=System: Manage Host Keytab+nsuniqueid=708bba65-14a611e5-8a48fd19-df27ff 01,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru ipaPermTargetFilter: (objectclass=ipahost) ipaPermRight: write ipaPermBindRuleType: permission ipaPermissionType: V2 ipaPermissionType: MANAGED ipaPermissionType: SYSTEM cn: System: Manage Host Keytab objectClass: ipapermission objectClass: top objectClass: groupofnames objectClass: ipapermissionv2 member: cn=Host Enrollment,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru member: cn=Host Administrators,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru ipaPermDefaultAttr: krbprincipalkey ipaPermDefaultAttr: krblastpwdchange ipaPermLocation: cn=computers,cn=accounts,dc=unix,dc=megafon,dc=ru # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 No other servers in IPA domain have such duplicates. WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: freeipa-users-boun...@redhat.com [mailto:freeipa-users-boun...@redhat.com] On Behalf Of Ludwig Krispenz Sent: Tuesday, June 16, 2015 3:52 PM To: freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts On 06/16/2015 11:42 AM, Alexander Frolushkin wrote: Hello. Just to remind if somebody still not familiar with our IPA installation :) We currently have 18 IPA servers in domain, on 8 sites in different regions across the Russia. And now, our new problem. Regularly we getting a nsds5ReplConflict records on some of our servers, very often on servers from specific site. Usually it is simply a doubles and we can remove the renamed change to get everything back. But why do we have them at all? May be someone could explain, how we can detect the cause of this replication conflicts? if you are talking about having two duplicate entries, one: uid=x,suffix one: nsuniqueid=+uid=x,suffix these entries appear if the entry uid=x was added, simultaneously, on two servers. I think this can happen if a client tries to add an entry and if it doesn't get a response in some time retries on another server. to find out which client this is you need to check on which servers the entries were originally added and then see which client was doing it Sometime it is moderately harmful, because, for example HBAC stops working on specific server while doubles still present. Thanks in forward... WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 Информация в этом сообщении предназначена исключительно для конкретных лиц, которым она адресована. В сообщении может содержаться конфиденциальная информация, которая не может быть раскрыта или использована кем-либо, кроме адресатов. Если вы не адресат этого сообщения, то использование, переадресация, копирование или распространение содержания сообщения или его части незаконно и запрещено. Если Вы получили это сообщение ошибочно, пожалуйста, незамедлительно сообщите отправителю об этом и удалите со всем содержимым само сообщение и любые возможные его копии и приложения. The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. The contents may not be disclosed or used by anyone other than the addressee. If you are not the intended recipient(s), any use, disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. If you have received
Re: [Freeipa-users] replication conflicts
Hi, this is really strange, if these conflict entries get created they should be the same on all servers. could you repeat the two searches requesting the attribute nscpentrywsi (you have to do it as directory manager, and add -o ldif-wrap=no), it could give info when and where these entries were created. Ludwig On 06/17/2015 08:13 AM, Alexander Frolushkin wrote: Hello. Anotherexample. Today appeared on servers of different site. Original LDIF: # extended LDIF # # LDAPv3 # base cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru with scope subtree # filter: (objectclass=*) # requesting: ALL # # System: Manage Host Keytab, permissions, pbac, unix.megafon.ru dn: cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc =ru ipaPermTargetFilter: (objectclass=ipahost) ipaPermRight: write ipaPermBindRuleType: permission ipaPermissionType: V2 ipaPermissionType: MANAGED ipaPermissionType: SYSTEM cn: System: Manage Host Keytab objectClass: ipapermission objectClass: top objectClass: groupofnames objectClass: ipapermissionv2 member: cn=Host Enrollment,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru member: cn=Host Administrators,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru ipaPermDefaultAttr: krbprincipalkey ipaPermDefaultAttr: krblastpwdchange ipaPermLocation: cn=computers,cn=accounts,dc=unix,dc=megafon,dc=ru # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 Duplicate: # extended LDIF # # LDAPv3 # base cn=System: Manage Host Keytab+nsuniqueid=708bba65-14a611e5-8a48fd19-df27ff01,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru with scope subtree # filter: (objectclass=*) # requesting: ALL # # System: Manage Host Keytab + 708bba65-14a611e5-8a48fd19-df27ff01, permissio ns, pbac, unix.megafon.ru dn: cn=System: Manage Host Keytab+nsuniqueid=708bba65-14a611e5-8a48fd19-df27ff 01,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru ipaPermTargetFilter: (objectclass=ipahost) ipaPermRight: write ipaPermBindRuleType: permission ipaPermissionType: V2 ipaPermissionType: MANAGED ipaPermissionType: SYSTEM cn: System: Manage Host Keytab objectClass: ipapermission objectClass: top objectClass: groupofnames objectClass: ipapermissionv2 member: cn=Host Enrollment,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru member: cn=Host Administrators,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru ipaPermDefaultAttr: krbprincipalkey ipaPermDefaultAttr: krblastpwdchange ipaPermLocation: cn=computers,cn=accounts,dc=unix,dc=megafon,dc=ru # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 No other servers in IPA domain have such duplicates. WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 *From:*freeipa-users-boun...@redhat.com [mailto:freeipa-users-boun...@redhat.com] *On Behalf Of *Ludwig Krispenz *Sent:* Tuesday, June 16, 2015 3:52 PM *To:* freeipa-users@redhat.com *Subject:* Re: [Freeipa-users] replication conflicts On 06/16/2015 11:42 AM, Alexander Frolushkin wrote: Hello. Just to remind if somebody still not familiar with our IPA installation J We currently have 18 IPA servers in domain, on 8 sites in different regions across the Russia. And now, our new problem. Regularly we getting a nsds5ReplConflict records on some of our servers, very often on servers from specific site. Usually it is simply a doubles and we can remove the renamed change to get everything back. But why do we have them at all? May be someone could explain, how we can detect the cause of this replication conflicts? if you are talking about having two duplicate entries, one: uid=x,suffix one: nsuniqueid=+uid=x,suffix these entries appear if the entry uid=x was added, simultaneously, on two servers. I think this can happen if a client tries to add an entry and if it doesn't get a response in some time retries on another server. to find out which client this is you need to check on which servers the entries were originally added and then see which client was doing it Sometime it is moderately harmful, because, for example HBAC stops working on specific server while doubles still present. Thanks in forward... WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 Информация в этом сообщении предназначена исключительно для конкретных лиц, которым она адресована. В сообщении может содержаться конфиденциальная информация, которая не может быть раскрыта или использована кем-либо, кроме адресатов. Если вы не адресат этого сообщения, то использование, переадресация, копирование или распространение содержания сообщения или его части незаконно и запрещено. Если Вы получили это сообщение ошибочно, пожалуйста, незамедлительно сообщите отправителю об этом и удалите со всем содержимым само сообщение и любые
Re: [Freeipa-users] replication conflicts
Hi, you did send the data directly to me, maybe not wanting to share them to everyone. I'll continue discussion here, trying to be careful. The good entry was created in April on replica 12 0x0c createTimestamp;vucsn-5524d42b0067000c: 20150408070720Z the nsuniqueid entry was created today on replica 26 0x1a createTimestamp;vucsn-5580f321001a: 20150617040801Z if the original entry would have existed on replica26 the new add should have been rejected, if it was not there the question is why. Do you have any additional info on replica 26, when was it created, was it disconnected for some time ?? Ludwig On 06/17/2015 08:13 AM, Alexander Frolushkin wrote: Hello. Anotherexample. Today appeared on servers of different site. Original LDIF: # extended LDIF # # LDAPv3 # base cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru with scope subtree # filter: (objectclass=*) # requesting: ALL # # System: Manage Host Keytab, permissions, pbac, unix.megafon.ru dn: cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc =ru ipaPermTargetFilter: (objectclass=ipahost) ipaPermRight: write ipaPermBindRuleType: permission ipaPermissionType: V2 ipaPermissionType: MANAGED ipaPermissionType: SYSTEM cn: System: Manage Host Keytab objectClass: ipapermission objectClass: top objectClass: groupofnames objectClass: ipapermissionv2 member: cn=Host Enrollment,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru member: cn=Host Administrators,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru ipaPermDefaultAttr: krbprincipalkey ipaPermDefaultAttr: krblastpwdchange ipaPermLocation: cn=computers,cn=accounts,dc=unix,dc=megafon,dc=ru # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 Duplicate: # extended LDIF # # LDAPv3 # base cn=System: Manage Host Keytab+nsuniqueid=708bba65-14a611e5-8a48fd19-df27ff01,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru with scope subtree # filter: (objectclass=*) # requesting: ALL # # System: Manage Host Keytab + 708bba65-14a611e5-8a48fd19-df27ff01, permissio ns, pbac, unix.megafon.ru dn: cn=System: Manage Host Keytab+nsuniqueid=708bba65-14a611e5-8a48fd19-df27ff 01,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru ipaPermTargetFilter: (objectclass=ipahost) ipaPermRight: write ipaPermBindRuleType: permission ipaPermissionType: V2 ipaPermissionType: MANAGED ipaPermissionType: SYSTEM cn: System: Manage Host Keytab objectClass: ipapermission objectClass: top objectClass: groupofnames objectClass: ipapermissionv2 member: cn=Host Enrollment,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru member: cn=Host Administrators,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru ipaPermDefaultAttr: krbprincipalkey ipaPermDefaultAttr: krblastpwdchange ipaPermLocation: cn=computers,cn=accounts,dc=unix,dc=megafon,dc=ru # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 No other servers in IPA domain have such duplicates. WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 *From:*freeipa-users-boun...@redhat.com [mailto:freeipa-users-boun...@redhat.com] *On Behalf Of *Ludwig Krispenz *Sent:* Tuesday, June 16, 2015 3:52 PM *To:* freeipa-users@redhat.com *Subject:* Re: [Freeipa-users] replication conflicts On 06/16/2015 11:42 AM, Alexander Frolushkin wrote: Hello. Just to remind if somebody still not familiar with our IPA installation J We currently have 18 IPA servers in domain, on 8 sites in different regions across the Russia. And now, our new problem. Regularly we getting a nsds5ReplConflict records on some of our servers, very often on servers from specific site. Usually it is simply a doubles and we can remove the renamed change to get everything back. But why do we have them at all? May be someone could explain, how we can detect the cause of this replication conflicts? if you are talking about having two duplicate entries, one: uid=x,suffix one: nsuniqueid=+uid=x,suffix these entries appear if the entry uid=x was added, simultaneously, on two servers. I think this can happen if a client tries to add an entry and if it doesn't get a response in some time retries on another server. to find out which client this is you need to check on which servers the entries were originally added and then see which client was doing it Sometime it is moderately harmful, because, for example HBAC stops working on specific server while doubles still present. Thanks in forward... WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 Информация в этом сообщении предназначена исключительно для конкретных лиц, которым она адресована. В сообщении может содержаться конфиденциальная информация, которая не может быть раскрыта или использована кем-либо, кроме адресатов. Если
Re: [Freeipa-users] replication conflicts
This is correct, thank you for understanding and for helping! Replica with id 26 was created today, this is our new server which was included in domain just a few hours ago. Looks like this dup came right after this new replica creation. WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: Ludwig Krispenz [mailto:lkris...@redhat.com] Sent: Wednesday, June 17, 2015 2:58 PM To: Alexander Frolushkin (SIB) Cc: freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts Hi, you did send the data directly to me, maybe not wanting to share them to everyone. I'll continue discussion here, trying to be careful. The good entry was created in April on replica 12 0x0c createTimestamp;vucsn-5524d42b0067000c: 20150408070720Z the nsuniqueid entry was created today on replica 26 0x1a createTimestamp;vucsn-5580f321001a: 20150617040801Z if the original entry would have existed on replica26 the new add should have been rejected, if it was not there the question is why. Do you have any additional info on replica 26, when was it created, was it disconnected for some time ?? Ludwig On 06/17/2015 08:13 AM, Alexander Frolushkin wrote: Hello. Another example. Today appeared on servers of different site. Original LDIF: # extended LDIF # # LDAPv3 # base cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru with scope subtree # filter: (objectclass=*) # requesting: ALL # # System: Manage Host Keytab, permissions, pbac, unix.megafon.ru dn: cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc =ru ipaPermTargetFilter: (objectclass=ipahost) ipaPermRight: write ipaPermBindRuleType: permission ipaPermissionType: V2 ipaPermissionType: MANAGED ipaPermissionType: SYSTEM cn: System: Manage Host Keytab objectClass: ipapermission objectClass: top objectClass: groupofnames objectClass: ipapermissionv2 member: cn=Host Enrollment,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru member: cn=Host Administrators,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru ipaPermDefaultAttr: krbprincipalkey ipaPermDefaultAttr: krblastpwdchange ipaPermLocation: cn=computers,cn=accounts,dc=unix,dc=megafon,dc=ru # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 Duplicate: # extended LDIF # # LDAPv3 # base cn=System: Manage Host Keytab+nsuniqueid=708bba65-14a611e5-8a48fd19-df27ff01,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru with scope subtree # filter: (objectclass=*) # requesting: ALL # # System: Manage Host Keytab + 708bba65-14a611e5-8a48fd19-df27ff01, permissio ns, pbac, unix.megafon.ru dn: cn=System: Manage Host Keytab+nsuniqueid=708bba65-14a611e5-8a48fd19-df27ff 01,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru ipaPermTargetFilter: (objectclass=ipahost) ipaPermRight: write ipaPermBindRuleType: permission ipaPermissionType: V2 ipaPermissionType: MANAGED ipaPermissionType: SYSTEM cn: System: Manage Host Keytab objectClass: ipapermission objectClass: top objectClass: groupofnames objectClass: ipapermissionv2 member: cn=Host Enrollment,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru member: cn=Host Administrators,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru ipaPermDefaultAttr: krbprincipalkey ipaPermDefaultAttr: krblastpwdchange ipaPermLocation: cn=computers,cn=accounts,dc=unix,dc=megafon,dc=ru # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 No other servers in IPA domain have such duplicates. WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: freeipa-users-boun...@redhat.commailto:freeipa-users-boun...@redhat.com [mailto:freeipa-users-boun...@redhat.com] On Behalf Of Ludwig Krispenz Sent: Tuesday, June 16, 2015 3:52 PM To: freeipa-users@redhat.commailto:freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts On 06/16/2015 11:42 AM, Alexander Frolushkin wrote: Hello. Just to remind if somebody still not familiar with our IPA installation :) We currently have 18 IPA servers in domain, on 8 sites in different regions across the Russia. And now, our new problem. Regularly we getting a nsds5ReplConflict records on some of our servers, very often on servers from specific site. Usually it is simply a doubles and we can remove the renamed change to get everything back. But why do we have them at all? May be someone could explain, how we can detect the cause of this replication conflicts? if you are talking about having two duplicate entries, one: uid=x,suffix one: nsuniqueid=+uid=x,suffix these entries appear if the entry uid=x was added, simultaneously, on two servers. I think this can happen if a client tries to add an entry and if it doesn't get a response in some time retries on another server. to find out which client this is you need to check on which servers the entries were originally added and then see which client was doing it Sometime it is moderately harmful, because, for example HBAC stops working on specific
Re: [Freeipa-users] replication conflicts
On 06/17/2015 11:03 AM, Alexander Frolushkin wrote: This is correct, thank you for understanding and for helping! Replica with id 26 was created today, this is our new server which was included in domain just a few hours ago. Looks like this dup came right after this new replica creation. so on which servers does the nsuniqueid entry exist ? can you check for 5580f321001a in the access log of replica 26, then check the errro log around this time and eventually the replica install log WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 *From:*Ludwig Krispenz [mailto:lkris...@redhat.com] *Sent:* Wednesday, June 17, 2015 2:58 PM *To:* Alexander Frolushkin (SIB) *Cc:* freeipa-users@redhat.com *Subject:* Re: [Freeipa-users] replication conflicts Hi, you did send the data directly to me, maybe not wanting to share them to everyone. I'll continue discussion here, trying to be careful. The good entry was created in April on replica 12 0x0c createTimestamp;vucsn-5524d42b0067000c: 20150408070720Z the nsuniqueid entry was created today on replica 26 0x1a createTimestamp;vucsn-5580f321001a: 20150617040801Z if the original entry would have existed on replica26 the new add should have been rejected, if it was not there the question is why. Do you have any additional info on replica 26, when was it created, was it disconnected for some time ?? Ludwig On 06/17/2015 08:13 AM, Alexander Frolushkin wrote: Hello. Another example. Today appeared on servers of different site. Original LDIF: # extended LDIF # # LDAPv3 # base cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru with scope subtree # filter: (objectclass=*) # requesting: ALL # # System: Manage Host Keytab, permissions, pbac, unix.megafon.ru dn: cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc =ru ipaPermTargetFilter: (objectclass=ipahost) ipaPermRight: write ipaPermBindRuleType: permission ipaPermissionType: V2 ipaPermissionType: MANAGED ipaPermissionType: SYSTEM cn: System: Manage Host Keytab objectClass: ipapermission objectClass: top objectClass: groupofnames objectClass: ipapermissionv2 member: cn=Host Enrollment,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru member: cn=Host Administrators,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru ipaPermDefaultAttr: krbprincipalkey ipaPermDefaultAttr: krblastpwdchange ipaPermLocation: cn=computers,cn=accounts,dc=unix,dc=megafon,dc=ru # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 Duplicate: # extended LDIF # # LDAPv3 # base cn=System: Manage Host Keytab+nsuniqueid=708bba65-14a611e5-8a48fd19-df27ff01,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru with scope subtree # filter: (objectclass=*) # requesting: ALL # # System: Manage Host Keytab + 708bba65-14a611e5-8a48fd19-df27ff01, permissio ns, pbac, unix.megafon.ru dn: cn=System: Manage Host Keytab+nsuniqueid=708bba65-14a611e5-8a48fd19-df27ff 01,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru ipaPermTargetFilter: (objectclass=ipahost) ipaPermRight: write ipaPermBindRuleType: permission ipaPermissionType: V2 ipaPermissionType: MANAGED ipaPermissionType: SYSTEM cn: System: Manage Host Keytab objectClass: ipapermission objectClass: top objectClass: groupofnames objectClass: ipapermissionv2 member: cn=Host Enrollment,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru member: cn=Host Administrators,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru ipaPermDefaultAttr: krbprincipalkey ipaPermDefaultAttr: krblastpwdchange ipaPermLocation: cn=computers,cn=accounts,dc=unix,dc=megafon,dc=ru # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 No other servers in IPA domain have such duplicates. WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 *From:*freeipa-users-boun...@redhat.com mailto:freeipa-users-boun...@redhat.com [mailto:freeipa-users-boun...@redhat.com] *On Behalf Of *Ludwig Krispenz *Sent:* Tuesday, June 16, 2015 3:52 PM *To:* freeipa-users@redhat.com mailto:freeipa-users@redhat.com *Subject:* Re: [Freeipa-users] replication conflicts On 06/16/2015 11:42 AM, Alexander Frolushkin wrote: Hello. Just to remind if somebody still not familiar with our IPA installation J We currently have 18 IPA servers in domain, on 8 sites in different regions across the Russia. And now, our new problem. Regularly we getting a nsds5ReplConflict records on some of our servers, very often on servers from specific site. Usually
Re: [Freeipa-users] replication conflicts
# grep conn=237 ./access [17/Jun/2015:14:37:03 +0600] conn=237 fd=71 slot=71 connection from 10.99.75.82 to 10.61.8.2 [17/Jun/2015:14:37:03 +0600] conn=237 op=0 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:14:37:03 +0600] conn=237 op=0 RESULT err=14 tag=97 nentries=0 etime=0, SASL bind in progress [17/Jun/2015:14:37:03 +0600] conn=237 op=1 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:14:37:03 +0600] conn=237 op=1 RESULT err=14 tag=97 nentries=0 etime=0, SASL bind in progress [17/Jun/2015:14:37:03 +0600] conn=237 op=2 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:14:37:03 +0600] conn=237 op=2 RESULT err=0 tag=97 nentries=0 etime=0 dn=krbprincipalname=ldap/msk-rhidm-03.unix.megafon...@unix.megafon.ru,cn=services,cn=accounts,dc=unix,dc=megafon,dc=ru [17/Jun/2015:14:37:03 +0600] conn=237 op=3 SRCH base= scope=0 filter=(objectClass=*) attrs=supportedControl supportedExtension [17/Jun/2015:14:37:03 +0600] conn=237 op=3 RESULT err=0 tag=101 nentries=1 etime=0 [17/Jun/2015:14:37:03 +0600] conn=237 op=4 SRCH base= scope=0 filter=(objectClass=*) attrs=supportedControl supportedExtension [17/Jun/2015:14:37:03 +0600] conn=237 op=4 RESULT err=0 tag=101 nentries=1 etime=0 [17/Jun/2015:14:37:03 +0600] conn=237 op=5 EXT oid=2.16.840.1.113730.3.5.12 name=replication-multimaster-extop [17/Jun/2015:14:37:03 +0600] conn=237 op=5 RESULT err=0 tag=120 nentries=0 etime=0 [17/Jun/2015:14:37:04 +0600] conn=237 op=6 EXT oid=2.16.840.1.113730.3.5.5 name=Netscape Replication End Session [17/Jun/2015:14:37:04 +0600] conn=237 op=6 RESULT err=0 tag=120 nentries=0 etime=0 [17/Jun/2015:14:37:05 +0600] conn=237 op=7 EXT oid=2.16.840.1.113730.3.5.12 name=replication-multimaster-extop [17/Jun/2015:14:37:05 +0600] conn=237 op=7 RESULT err=0 tag=120 nentries=0 etime=0 [17/Jun/2015:14:37:07 +0600] conn=237 op=8 EXT oid=2.16.840.1.113730.3.5.5 name=Netscape Replication End Session WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: freeipa-users-boun...@redhat.com [mailto:freeipa-users-boun...@redhat.com] On Behalf Of Ludwig Krispenz Sent: Wednesday, June 17, 2015 3:55 PM To: freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts On 06/17/2015 11:52 AM, Ludwig Krispenz wrote: On 06/17/2015 11:45 AM, thierry bordaz wrote: On 06/17/2015 11:22 AM, Alexander Frolushkin wrote: This was a usual ipa-replica-install --setup-ca --setup-dns and after that ipa-adtrust-install. No DEL found: # grep cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru ./access [17/Jun/2015:10:08:01 +0600] conn=2 op=89 SRCH base=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru scope=0 filter=(objectClass=*) attrs=ipaPermRight ipaPermTargetFilter ipaPermBindRuleType ipaPermissionType cn objectClass memberOf member ipaPermTarget ipaPermDefaultAttr ipaPermLocation ipaPermIncludedAttr ipaPermExcludedAttr [17/Jun/2015:10:08:01 +0600] conn=2 op=91 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru There is something I miss. conn=2 op=91 was a direct update on replica26 (not replicated) because it received its own CSN=5580f321001a. But it created a conflict entry, so at that time it existed the same entry (the one created 20150408070720Z) . So the direct update should have been rejected. I think the search in op=89 did not return an entry, so it was added in op 91, that seems to be ok, but then 4 hrs later there is conn=237 adding it again. Alexander, could you get the complete 'conn=237 op=93' and also the start of conn 293, to show where teh connection comes from of course conn=237 Would you check if the replicaID=26 is unique in the topology (list-ruv for example) ? [17/Jun/2015:14:39:46 +0600] conn=237 op=93 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru It is also possible this entry on affected servers was previously duplicated and not correctly managed to delete (more recent dup was deleted). Is there any natural way to fix such issues? Maybe ipa-replica-manage force-sync, or ipa-replica-manage re-initialize on affected site servers from normal servers could help? WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: thierry bordaz [mailto:tbor...@redhat.com] Sent: Wednesday, June 17, 2015 3:15 PM To: Alexander Frolushkin (SIB) Cc: 'Ludwig Krispenz'; freeipa-users@redhat.commailto:freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts Hello Alexander, How did you initialize that new replica 26. Either 'cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru' was not part of the total init data, or a DEL of that entry happened on replica 26 (before a new ADD) but the DEL was not replicated to replica12. Would you check in replica26 access logs if that entry was deleted ? thanks theirry On 06/17/2015 11:03 AM, Alexander Frolushkin wrote: This is correct, thank
Re: [Freeipa-users] replication conflicts
This is not a good news, because replica id 20 is not exist for a some days already. It was recreated and now have id 23 WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: thierry bordaz [mailto:tbor...@redhat.com] Sent: Wednesday, June 17, 2015 4:10 PM To: Alexander Frolushkin (SIB) Cc: 'Ludwig Krispenz'; freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts On 06/17/2015 11:56 AM, Alexander Frolushkin wrote: Will this be enough? # grep conn=237 op=93 ./access [17/Jun/2015:14:39:46 +0600] conn=237 op=93 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru [17/Jun/2015:14:39:46 +0600] conn=237 op=93 RESULT err=0 tag=105 nentries=0 etime=0 csn=555ac9360014 This operation is a replicated one and the CSN is from May 19th. So why a replica (26) created today was initialized without that entry ? This updates was originated from replica20. Was it stopped and restarted recently ? # grep conn=293 ./access [17/Jun/2015:15:33:04 +0600] conn=293 fd=75 slot=75 connection from 10.99.75.82 to 10.61.8.2 [17/Jun/2015:15:33:04 +0600] conn=293 op=0 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:15:33:04 +0600] conn=293 op=0 RESULT err=14 tag=97 nentries=0 etime=0, SASL bind in progress [17/Jun/2015:15:33:04 +0600] conn=293 op=1 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:15:33:04 +0600] conn=293 op=1 RESULT err=14 tag=97 nentries=0 etime=0, SASL bind in progress [17/Jun/2015:15:33:04 +0600] conn=293 op=2 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:15:33:04 +0600] conn=293 op=2 RESULT err=0 tag=97 nentries=0 etime=0 dn=krbprincipalname=ldap/msk-rhidm-03.unix.megafon...@unix.megafon.ru,cn=services,cn=accounts,dc=unix,dc=megafon,dc=rumailto:krbprincipalname=ldap/msk-rhidm-03.unix.megafon...@unix.megafon.ru,cn=services,cn=accounts,dc=unix,dc=megafon,dc=ru WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: Ludwig Krispenz [mailto:lkris...@redhat.com] Sent: Wednesday, June 17, 2015 3:53 PM To: thierry bordaz Cc: Alexander Frolushkin (SIB); freeipa-users@redhat.commailto:freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts On 06/17/2015 11:45 AM, thierry bordaz wrote: On 06/17/2015 11:22 AM, Alexander Frolushkin wrote: This was a usual ipa-replica-install --setup-ca --setup-dns and after that ipa-adtrust-install. No DEL found: # grep cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru ./access [17/Jun/2015:10:08:01 +0600] conn=2 op=89 SRCH base=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru scope=0 filter=(objectClass=*) attrs=ipaPermRight ipaPermTargetFilter ipaPermBindRuleType ipaPermissionType cn objectClass memberOf member ipaPermTarget ipaPermDefaultAttr ipaPermLocation ipaPermIncludedAttr ipaPermExcludedAttr [17/Jun/2015:10:08:01 +0600] conn=2 op=91 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru There is something I miss. conn=2 op=91 was a direct update on replica26 (not replicated) because it received its own CSN=5580f321001a. But it created a conflict entry, so at that time it existed the same entry (the one created 20150408070720Z) . So the direct update should have been rejected. I think the search in op=89 did not return an entry, so it was added in op 91, that seems to be ok, but then 4 hrs later there is conn=237 adding it again. Alexander, could you get the complete 'conn=237 op=93' and also the start of conn 293, to show where teh connection comes from Would you check if the replicaID=26 is unique in the topology (list-ruv for example) ? [17/Jun/2015:14:39:46 +0600] conn=237 op=93 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru It is also possible this entry on affected servers was previously duplicated and not correctly managed to delete (more recent dup was deleted). Is there any natural way to fix such issues? Maybe ipa-replica-manage force-sync, or ipa-replica-manage re-initialize on affected site servers from normal servers could help? WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: thierry bordaz [mailto:tbor...@redhat.com] Sent: Wednesday, June 17, 2015 3:15 PM To: Alexander Frolushkin (SIB) Cc: 'Ludwig Krispenz'; freeipa-users@redhat.commailto:freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts Hello Alexander, How did you initialize that new replica 26. Either 'cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru' was not part of the total init data, or a DEL of that entry happened on replica 26 (before a new ADD) but the DEL was not replicated to replica12. Would you check in replica26 access logs if that entry was deleted ? thanks theirry On 06/17/2015 11:03 AM, Alexander Frolushkin wrote: This is correct, thank you for understanding and for helping! Replica with id 26 was created
Re: [Freeipa-users] replication conflicts
Will this be enough? # grep conn=237 op=93 ./access [17/Jun/2015:14:39:46 +0600] conn=237 op=93 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru [17/Jun/2015:14:39:46 +0600] conn=237 op=93 RESULT err=0 tag=105 nentries=0 etime=0 csn=555ac9360014 # grep conn=293 ./access [17/Jun/2015:15:33:04 +0600] conn=293 fd=75 slot=75 connection from 10.99.75.82 to 10.61.8.2 [17/Jun/2015:15:33:04 +0600] conn=293 op=0 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:15:33:04 +0600] conn=293 op=0 RESULT err=14 tag=97 nentries=0 etime=0, SASL bind in progress [17/Jun/2015:15:33:04 +0600] conn=293 op=1 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:15:33:04 +0600] conn=293 op=1 RESULT err=14 tag=97 nentries=0 etime=0, SASL bind in progress [17/Jun/2015:15:33:04 +0600] conn=293 op=2 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:15:33:04 +0600] conn=293 op=2 RESULT err=0 tag=97 nentries=0 etime=0 dn=krbprincipalname=ldap/msk-rhidm-03.unix.megafon...@unix.megafon.ru,cn=services,cn=accounts,dc=unix,dc=megafon,dc=ru WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: Ludwig Krispenz [mailto:lkris...@redhat.com] Sent: Wednesday, June 17, 2015 3:53 PM To: thierry bordaz Cc: Alexander Frolushkin (SIB); freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts On 06/17/2015 11:45 AM, thierry bordaz wrote: On 06/17/2015 11:22 AM, Alexander Frolushkin wrote: This was a usual ipa-replica-install --setup-ca --setup-dns and after that ipa-adtrust-install. No DEL found: # grep cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru ./access [17/Jun/2015:10:08:01 +0600] conn=2 op=89 SRCH base=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru scope=0 filter=(objectClass=*) attrs=ipaPermRight ipaPermTargetFilter ipaPermBindRuleType ipaPermissionType cn objectClass memberOf member ipaPermTarget ipaPermDefaultAttr ipaPermLocation ipaPermIncludedAttr ipaPermExcludedAttr [17/Jun/2015:10:08:01 +0600] conn=2 op=91 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru There is something I miss. conn=2 op=91 was a direct update on replica26 (not replicated) because it received its own CSN=5580f321001a. But it created a conflict entry, so at that time it existed the same entry (the one created 20150408070720Z) . So the direct update should have been rejected. I think the search in op=89 did not return an entry, so it was added in op 91, that seems to be ok, but then 4 hrs later there is conn=237 adding it again. Alexander, could you get the complete 'conn=237 op=93' and also the start of conn 293, to show where teh connection comes from Would you check if the replicaID=26 is unique in the topology (list-ruv for example) ? [17/Jun/2015:14:39:46 +0600] conn=237 op=93 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru It is also possible this entry on affected servers was previously duplicated and not correctly managed to delete (more recent dup was deleted). Is there any natural way to fix such issues? Maybe ipa-replica-manage force-sync, or ipa-replica-manage re-initialize on affected site servers from normal servers could help? WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: thierry bordaz [mailto:tbor...@redhat.com] Sent: Wednesday, June 17, 2015 3:15 PM To: Alexander Frolushkin (SIB) Cc: 'Ludwig Krispenz'; freeipa-users@redhat.commailto:freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts Hello Alexander, How did you initialize that new replica 26. Either 'cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru' was not part of the total init data, or a DEL of that entry happened on replica 26 (before a new ADD) but the DEL was not replicated to replica12. Would you check in replica26 access logs if that entry was deleted ? thanks theirry On 06/17/2015 11:03 AM, Alexander Frolushkin wrote: This is correct, thank you for understanding and for helping! Replica with id 26 was created today, this is our new server which was included in domain just a few hours ago. Looks like this dup came right after this new replica creation. WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: Ludwig Krispenz [mailto:lkris...@redhat.com] Sent: Wednesday, June 17, 2015 2:58 PM To: Alexander Frolushkin (SIB) Cc: freeipa-users@redhat.commailto:freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts Hi, you did send the data directly to me, maybe not wanting to share them to everyone. I'll continue discussion here, trying to be careful. The good entry was created in April on replica 12 0x0c createTimestamp;vucsn-5524d42b0067000c: 20150408070720Z the nsuniqueid entry was created today on replica 26 0x1a createTimestamp;vucsn-5580f321001a: 20150617040801Z if the original entry
Re: [Freeipa-users] replication conflicts
conn=237 is from 10.99.75.82 which replica is this ? msk-rhidm-03.unix.megafon.ru:389: 10 On 06/17/2015 12:13 PM, Alexander Frolushkin wrote: This is not a good news, because replica id 20 is not exist for a some days already. It was recreated and now have id 23 WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: thierry bordaz [mailto:tbor...@redhat.com] Sent: Wednesday, June 17, 2015 4:10 PM To: Alexander Frolushkin (SIB) Cc: 'Ludwig Krispenz'; freeipa-users@redhat.commailto:freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts On 06/17/2015 11:56 AM, Alexander Frolushkin wrote: Will this be enough? # grep conn=237 op=93 ./access [17/Jun/2015:14:39:46 +0600] conn=237 op=93 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru [17/Jun/2015:14:39:46 +0600] conn=237 op=93 RESULT err=0 tag=105 nentries=0 etime=0 csn=555ac9360014 This operation is a replicated one and the CSN is from May 19th. So why a replica (26) created today was initialized without that entry ? This updates was originated from replica20. Was it stopped and restarted recently ? # grep conn=293 ./access [17/Jun/2015:15:33:04 +0600] conn=293 fd=75 slot=75 connection from 10.99.75.82 to 10.61.8.2 [17/Jun/2015:15:33:04 +0600] conn=293 op=0 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:15:33:04 +0600] conn=293 op=0 RESULT err=14 tag=97 nentries=0 etime=0, SASL bind in progress [17/Jun/2015:15:33:04 +0600] conn=293 op=1 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:15:33:04 +0600] conn=293 op=1 RESULT err=14 tag=97 nentries=0 etime=0, SASL bind in progress [17/Jun/2015:15:33:04 +0600] conn=293 op=2 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:15:33:04 +0600] conn=293 op=2 RESULT err=0 tag=97 nentries=0 etime=0 dn=krbprincipalname=ldap/msk-rhidm-03.unix.megafon...@unix.megafon.ru,cn=services,cn=accounts,dc=unix,dc=megafon,dc=rumailto:krbprincipalname=ldap/msk-rhidm-03.unix.megafon...@unix.megafon.ru,cn=services,cn=accounts,dc=unix,dc=megafon,dc=ru WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: Ludwig Krispenz [mailto:lkris...@redhat.com] Sent: Wednesday, June 17, 2015 3:53 PM To: thierry bordaz Cc: Alexander Frolushkin (SIB); freeipa-users@redhat.commailto:freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts On 06/17/2015 11:45 AM, thierry bordaz wrote: On 06/17/2015 11:22 AM, Alexander Frolushkin wrote: This was a usual ipa-replica-install --setup-ca --setup-dns and after that ipa-adtrust-install. No DEL found: # grep cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru ./access [17/Jun/2015:10:08:01 +0600] conn=2 op=89 SRCH base=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru scope=0 filter=(objectClass=*) attrs=ipaPermRight ipaPermTargetFilter ipaPermBindRuleType ipaPermissionType cn objectClass memberOf member ipaPermTarget ipaPermDefaultAttr ipaPermLocation ipaPermIncludedAttr ipaPermExcludedAttr [17/Jun/2015:10:08:01 +0600] conn=2 op=91 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru There is something I miss. conn=2 op=91 was a direct update on replica26 (not replicated) because it received its own CSN=5580f321001a. But it created a conflict entry, so at that time it existed the same entry (the one created 20150408070720Z) . So the direct update should have been rejected. I think the search in op=89 did not return an entry, so it was added in op 91, that seems to be ok, but then 4 hrs later there is conn=237 adding it again. Alexander, could you get the complete 'conn=237 op=93' and also the start of conn 293, to show where teh connection comes from Would you check if the replicaID=26 is unique in the topology (list-ruv for example) ? [17/Jun/2015:14:39:46 +0600] conn=237 op=93 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru It is also possible this entry on affected servers was previously duplicated and not correctly managed to delete (more recent dup was deleted). Is there any natural way to fix such issues? Maybe ipa-replica-manage force-sync, or ipa-replica-manage re-initialize on affected site servers from normal servers could help? WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: thierry bordaz [mailto:tbor...@redhat.com] Sent: Wednesday, June 17, 2015 3:15 PM To: Alexander Frolushkin (SIB) Cc: 'Ludwig Krispenz'; freeipa-users@redhat.commailto:freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts Hello Alexander, How did you initialize that new replica 26. Either 'cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru' was not part of the total init data, or a DEL of that entry happened on replica 26 (before a new ADD) but the DEL was not replicated to replica12. Would you check in replica26 access logs if that entry
Re: [Freeipa-users] replication conflicts
conn=237 is from 10.99.75.82 which replica is this ? On 06/17/2015 12:13 PM, Alexander Frolushkin wrote: This is not a good news, because replica id 20 is not exist for a some days already. It was recreated and now have id 23 WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 *From:*thierry bordaz [mailto:tbor...@redhat.com] *Sent:* Wednesday, June 17, 2015 4:10 PM *To:* Alexander Frolushkin (SIB) *Cc:* 'Ludwig Krispenz'; freeipa-users@redhat.com *Subject:* Re: [Freeipa-users] replication conflicts On 06/17/2015 11:56 AM, Alexander Frolushkin wrote: Will this be enough? # grep conn=237 op=93 ./access [17/Jun/2015:14:39:46 +0600] conn=237 op=93 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru [17/Jun/2015:14:39:46 +0600] conn=237 op=93 RESULT err=0 tag=105 nentries=0 etime=0 csn=555ac9360014 This operation is a replicated one and the CSN is from May 19th. So why a replica (26) created today was initialized without that entry ? This updates was originated from replica20. Was it stopped and restarted recently ? # grep conn=293 ./access [17/Jun/2015:15:33:04 +0600] conn=293 fd=75 slot=75 connection from 10.99.75.82 to 10.61.8.2 [17/Jun/2015:15:33:04 +0600] conn=293 op=0 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:15:33:04 +0600] conn=293 op=0 RESULT err=14 tag=97 nentries=0 etime=0, SASL bind in progress [17/Jun/2015:15:33:04 +0600] conn=293 op=1 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:15:33:04 +0600] conn=293 op=1 RESULT err=14 tag=97 nentries=0 etime=0, SASL bind in progress [17/Jun/2015:15:33:04 +0600] conn=293 op=2 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:15:33:04 +0600] conn=293 op=2 RESULT err=0 tag=97 nentries=0 etime=0 dn=krbprincipalname=ldap/msk-rhidm-03.unix.megafon...@unix.megafon.ru,cn=services,cn=accounts,dc=unix,dc=megafon,dc=ru mailto:krbprincipalname=ldap/msk-rhidm-03.unix.megafon...@unix.megafon.ru,cn=services,cn=accounts,dc=unix,dc=megafon,dc=ru WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 *From:*Ludwig Krispenz [mailto:lkris...@redhat.com] *Sent:* Wednesday, June 17, 2015 3:53 PM *To:* thierry bordaz *Cc:* Alexander Frolushkin (SIB); freeipa-users@redhat.com mailto:freeipa-users@redhat.com *Subject:* Re: [Freeipa-users] replication conflicts On 06/17/2015 11:45 AM, thierry bordaz wrote: On 06/17/2015 11:22 AM, Alexander Frolushkin wrote: This was a usual ipa-replica-install --setup-ca --setup-dns and after that ipa-adtrust-install. No DEL found: # grep cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru ./access [17/Jun/2015:10:08:01 +0600] conn=2 op=89 SRCH base=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru scope=0 filter=(objectClass=*) attrs=ipaPermRight ipaPermTargetFilter ipaPermBindRuleType ipaPermissionType cn objectClass memberOf member ipaPermTarget ipaPermDefaultAttr ipaPermLocation ipaPermIncludedAttr ipaPermExcludedAttr [17/Jun/2015:10:08:01 +0600] conn=2 op=91 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru There is something I miss. conn=2 op=91 was a direct update on replica26 (not replicated) because it received its own CSN=5580f321001a. But it created a conflict entry, so at that time it existed the same entry (the one created 20150408070720Z) . So the direct update should have been rejected. I think the search in op=89 did not return an entry, so it was added in op 91, that seems to be ok, but then 4 hrs later there is conn=237 adding it again. Alexander, could you get the complete 'conn=237 op=93' and also the start of conn 293, to show where teh connection comes from Would you check if the replicaID=26 is unique in the topology (list-ruv for example) ? [17/Jun/2015:14:39:46 +0600] conn=237 op=93 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru It is also possible this entry on affected servers was previously duplicated and not correctly managed to delete (more recent dup was deleted). Is there any natural way to fix such issues? Maybe ipa-replica-manage force-sync, or ipa-replica-manage re-initialize on affected site servers from normal servers could help? WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 *From:*thierry bordaz [mailto:tbor...@redhat.com] *Sent:* Wednesday, June 17, 2015 3:15 PM *To:* Alexander Frolushkin (SIB) *Cc:* 'Ludwig Krispenz'; freeipa-users@redhat.com mailto:freeipa-users@redhat.com *Subject:* Re: [Freeipa-users] replication conflicts Hello Alexander, How did you initialize that new replica 26. Either 'cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru' was not part of the total init data
Re: [Freeipa-users] replication conflicts
+0600] - slapd shutting down - closing down internal subsystems and plugins [17/Jun/2015:10:08:03 +0600] NSMMReplicationPlugin - agmt=cn=meTomsk-rhidm-03.unix.megafon.ru (msk-rhidm-03:389): Warning: Attempting to release replica, but unable to receive endReplication exte nded operation response from the replica. Error -5 (Timed out) [17/Jun/2015:10:08:03 +0600] - Waiting for 4 database threads to stop [17/Jun/2015:10:08:04 +0600] - All database threads now stopped [17/Jun/2015:10:08:04 +0600] - slapd shutting down - freed 2 work q stack objects - freed 2 op stack objects [17/Jun/2015:10:08:04 +0600] - slapd stopped. [17/Jun/2015:10:08:06 +0600] SSL Initialization - Configured SSL version range: min: TLS1.0, max: TLS1.2 [17/Jun/2015:10:08:06 +0600] - SSL alert: Configured NSS Ciphers WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: Ludwig Krispenz [mailto:lkris...@redhat.com] Sent: Wednesday, June 17, 2015 3:19 PM To: Alexander Frolushkin (SIB) Cc: freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts On 06/17/2015 11:03 AM, Alexander Frolushkin wrote: This is correct, thank you for understanding and for helping! Replica with id 26 was created today, this is our new server which was included in domain just a few hours ago. Looks like this dup came right after this new replica creation. so on which servers does the nsuniqueid entry exist ? can you check for 5580f321001a in the access log of replica 26, then check the errro log around this time and eventually the replica install log WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: Ludwig Krispenz [mailto:lkris...@redhat.com] Sent: Wednesday, June 17, 2015 2:58 PM To: Alexander Frolushkin (SIB) Cc: freeipa-users@redhat.commailto:freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts Hi, you did send the data directly to me, maybe not wanting to share them to everyone. I'll continue discussion here, trying to be careful. The good entry was created in April on replica 12 0x0c createTimestamp;vucsn-5524d42b0067000c: 20150408070720Z the nsuniqueid entry was created today on replica 26 0x1a createTimestamp;vucsn-5580f321001a: 20150617040801Z if the original entry would have existed on replica26 the new add should have been rejected, if it was not there the question is why. Do you have any additional info on replica 26, when was it created, was it disconnected for some time ?? Ludwig On 06/17/2015 08:13 AM, Alexander Frolushkin wrote: Hello. Another example. Today appeared on servers of different site. Original LDIF: # extended LDIF # # LDAPv3 # base cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru with scope subtree # filter: (objectclass=*) # requesting: ALL # # System: Manage Host Keytab, permissions, pbac, unix.megafon.ru dn: cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc =ru ipaPermTargetFilter: (objectclass=ipahost) ipaPermRight: write ipaPermBindRuleType: permission ipaPermissionType: V2 ipaPermissionType: MANAGED ipaPermissionType: SYSTEM cn: System: Manage Host Keytab objectClass: ipapermission objectClass: top objectClass: groupofnames objectClass: ipapermissionv2 member: cn=Host Enrollment,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru member: cn=Host Administrators,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru ipaPermDefaultAttr: krbprincipalkey ipaPermDefaultAttr: krblastpwdchange ipaPermLocation: cn=computers,cn=accounts,dc=unix,dc=megafon,dc=ru # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 Duplicate: # extended LDIF # # LDAPv3 # base cn=System: Manage Host Keytab+nsuniqueid=708bba65-14a611e5-8a48fd19-df27ff01,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru with scope subtree # filter: (objectclass=*) # requesting: ALL # # System: Manage Host Keytab + 708bba65-14a611e5-8a48fd19-df27ff01, permissio ns, pbac, unix.megafon.ru dn: cn=System: Manage Host Keytab+nsuniqueid=708bba65-14a611e5-8a48fd19-df27ff 01,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru ipaPermTargetFilter: (objectclass=ipahost) ipaPermRight: write ipaPermBindRuleType: permission ipaPermissionType: V2 ipaPermissionType: MANAGED ipaPermissionType: SYSTEM cn: System: Manage Host Keytab objectClass: ipapermission objectClass: top objectClass: groupofnames objectClass: ipapermissionv2 member: cn=Host Enrollment,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru member: cn=Host Administrators,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru ipaPermDefaultAttr: krbprincipalkey ipaPermDefaultAttr: krblastpwdchange ipaPermLocation: cn=computers,cn=accounts,dc=unix,dc=megafon,dc=ru # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 No other servers in IPA domain have such duplicates. WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: freeipa-users-boun...@redhat.commailto:freeipa-users-boun...@redhat.com [mailto:freeipa-users-boun
Re: [Freeipa-users] replication conflicts
I'm pretty sure id 26 is unique
ipa-replica-manage list-ruv
Directory Manager password:
unable to decode: {replica 20} 555ac82600010014 55716e5700030014
unable to decode: {replica 24} 557fb7d400040018 557fb9a100100018
unable to decode: {replica 22} 5576b83e00010016 5576ba4b00020016
unable to decode: {replica 21} 5576ac960015 5576b52e00020015
dv-rhidm01.unix.megafon.ru:389: 17
cnt-rhidm01.unix.megafon.ru:389: 14
msk-rhidm-01.unix.megafon.ru:389: 9
sib-rhidm01.unix.megafon.ru:389: 4
cnt-rhidm02.unix.megafon.ru:389: 15
nw-rhidm02.unix.megafon.ru:389: 23
kvk-rhidm02.unix.megafon.ru:389: 26
dv-rhidm02.unix.megafon.ru:389: 18
vlg-rhidm01.unix.megafon.ru:389: 8
kvk-rhidm01.unix.megafon.ru:389: 25
msk-rhidm-02.unix.megafon.ru:389: 11
vlg-rhidm02.unix.megafon.ru:389: 13
url-rhidm01.unix.megafon.ru:389: 6
vlg-rhidm03.unix.megafon.ru:389: 12
sib-rhidm03.unix.megafon.ru:389: 5
nw-rhidm01.unix.megafon.ru:389: 19
url-rhidm02.unix.megafon.ru:389: 7
msk-rhidm-03.unix.megafon.ru:389: 10
sib-rhidm02.unix.megafon.ru:389: 3
Unable to decode - This is al all our existed in past replicas, which was
removed, but regularly appearing this way. We not yet found way to fix it
completely.
WBR,
Alexander Frolushkin
Cell +79232508764
Work +79232507764
From: thierry bordaz [mailto:tbor...@redhat.com]
Sent: Wednesday, June 17, 2015 3:46 PM
To: Alexander Frolushkin (SIB)
Cc: 'Ludwig Krispenz'; freeipa-users@redhat.com
Subject: Re: [Freeipa-users] replication conflicts
On 06/17/2015 11:22 AM, Alexander Frolushkin wrote:
This was a usual ipa-replica-install --setup-ca --setup-dns and after that
ipa-adtrust-install.
No DEL found:
# grep cn=System: Manage Host
Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru ./access
[17/Jun/2015:10:08:01 +0600] conn=2 op=89 SRCH base=cn=System: Manage Host
Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru scope=0
filter=(objectClass=*) attrs=ipaPermRight ipaPermTargetFilter
ipaPermBindRuleType ipaPermissionType cn objectClass memberOf member
ipaPermTarget ipaPermDefaultAttr ipaPermLocation ipaPermIncludedAttr
ipaPermExcludedAttr
[17/Jun/2015:10:08:01 +0600] conn=2 op=91 ADD dn=cn=System: Manage Host
Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru
There is something I miss. conn=2 op=91 was a direct update on replica26 (not
replicated) because it received its own CSN=5580f321001a. But it
created a conflict entry, so at that time it existed the same entry (the one
created 20150408070720Z) . So the direct update should have been rejected.
Would you check if the replicaID=26 is unique in the topology (list-ruv for
example) ?
[17/Jun/2015:14:39:46 +0600] conn=237 op=93 ADD dn=cn=System: Manage Host
Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru
It is also possible this entry on affected servers was previously duplicated
and not correctly managed to delete (more recent dup was deleted).
Is there any natural way to fix such issues? Maybe ipa-replica-manage
force-sync, or ipa-replica-manage re-initialize on affected site servers from
normal servers could help?
WBR,
Alexander Frolushkin
Cell +79232508764
Work +79232507764
From: thierry bordaz [mailto:tbor...@redhat.com]
Sent: Wednesday, June 17, 2015 3:15 PM
To: Alexander Frolushkin (SIB)
Cc: 'Ludwig Krispenz'; freeipa-users@redhat.commailto:freeipa-users@redhat.com
Subject: Re: [Freeipa-users] replication conflicts
Hello Alexander,
How did you initialize that new replica 26.
Either 'cn=System: Manage Host
Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru' was not part of the
total init data, or a DEL of that entry happened on replica 26 (before a new
ADD) but the DEL was not replicated to replica12.
Would you check in replica26 access logs if that entry was deleted ?
thanks
theirry
On 06/17/2015 11:03 AM, Alexander Frolushkin wrote:
This is correct, thank you for understanding and for helping!
Replica with id 26 was created today, this is our new server which was included
in domain just a few hours ago. Looks like this dup came right after this new
replica creation.
WBR,
Alexander Frolushkin
Cell +79232508764
Work +79232507764
From: Ludwig Krispenz [mailto:lkris...@redhat.com]
Sent: Wednesday, June 17, 2015 2:58 PM
To: Alexander Frolushkin (SIB)
Cc: freeipa-users@redhat.commailto:freeipa-users@redhat.com
Subject: Re: [Freeipa-users] replication conflicts
Hi,
you did send the data directly to me, maybe not wanting to share them to
everyone. I'll continue discussion here, trying to be careful.
The good entry was created in April on replica 12 0x0c
createTimestamp;vucsn-5524d42b0067000c: 20150408070720Z
the nsuniqueid entry was created today on replica 26 0x1a
createTimestamp;vucsn-5580f321001a: 20150617040801Z
if the original entry would have existed on replica26 the new add should have
been rejected, if it was not there the question is why.
Do you have any additional info on replica 26, when was it created, was it
disconnected
Re: [Freeipa-users] replication conflicts
This was a usual ipa-replica-install --setup-ca --setup-dns and after that ipa-adtrust-install. No DEL found: # grep cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru ./access [17/Jun/2015:10:08:01 +0600] conn=2 op=89 SRCH base=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru scope=0 filter=(objectClass=*) attrs=ipaPermRight ipaPermTargetFilter ipaPermBindRuleType ipaPermissionType cn objectClass memberOf member ipaPermTarget ipaPermDefaultAttr ipaPermLocation ipaPermIncludedAttr ipaPermExcludedAttr [17/Jun/2015:10:08:01 +0600] conn=2 op=91 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru [17/Jun/2015:14:39:46 +0600] conn=237 op=93 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru It is also possible this entry on affected servers was previously duplicated and not correctly managed to delete (more recent dup was deleted). Is there any natural way to fix such issues? Maybe ipa-replica-manage force-sync, or ipa-replica-manage re-initialize on affected site servers from normal servers could help? WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: thierry bordaz [mailto:tbor...@redhat.com] Sent: Wednesday, June 17, 2015 3:15 PM To: Alexander Frolushkin (SIB) Cc: 'Ludwig Krispenz'; freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts Hello Alexander, How did you initialize that new replica 26. Either 'cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru' was not part of the total init data, or a DEL of that entry happened on replica 26 (before a new ADD) but the DEL was not replicated to replica12. Would you check in replica26 access logs if that entry was deleted ? thanks theirry On 06/17/2015 11:03 AM, Alexander Frolushkin wrote: This is correct, thank you for understanding and for helping! Replica with id 26 was created today, this is our new server which was included in domain just a few hours ago. Looks like this dup came right after this new replica creation. WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: Ludwig Krispenz [mailto:lkris...@redhat.com] Sent: Wednesday, June 17, 2015 2:58 PM To: Alexander Frolushkin (SIB) Cc: freeipa-users@redhat.commailto:freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts Hi, you did send the data directly to me, maybe not wanting to share them to everyone. I'll continue discussion here, trying to be careful. The good entry was created in April on replica 12 0x0c createTimestamp;vucsn-5524d42b0067000c: 20150408070720Z the nsuniqueid entry was created today on replica 26 0x1a createTimestamp;vucsn-5580f321001a: 20150617040801Z if the original entry would have existed on replica26 the new add should have been rejected, if it was not there the question is why. Do you have any additional info on replica 26, when was it created, was it disconnected for some time ?? Ludwig On 06/17/2015 08:13 AM, Alexander Frolushkin wrote: Hello. Another example. Today appeared on servers of different site. Original LDIF: # extended LDIF # # LDAPv3 # base cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru with scope subtree # filter: (objectclass=*) # requesting: ALL # # System: Manage Host Keytab, permissions, pbac, unix.megafon.ru dn: cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc =ru ipaPermTargetFilter: (objectclass=ipahost) ipaPermRight: write ipaPermBindRuleType: permission ipaPermissionType: V2 ipaPermissionType: MANAGED ipaPermissionType: SYSTEM cn: System: Manage Host Keytab objectClass: ipapermission objectClass: top objectClass: groupofnames objectClass: ipapermissionv2 member: cn=Host Enrollment,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru member: cn=Host Administrators,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru ipaPermDefaultAttr: krbprincipalkey ipaPermDefaultAttr: krblastpwdchange ipaPermLocation: cn=computers,cn=accounts,dc=unix,dc=megafon,dc=ru # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 Duplicate: # extended LDIF # # LDAPv3 # base cn=System: Manage Host Keytab+nsuniqueid=708bba65-14a611e5-8a48fd19-df27ff01,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru with scope subtree # filter: (objectclass=*) # requesting: ALL # # System: Manage Host Keytab + 708bba65-14a611e5-8a48fd19-df27ff01, permissio ns, pbac, unix.megafon.ru dn: cn=System: Manage Host Keytab+nsuniqueid=708bba65-14a611e5-8a48fd19-df27ff 01,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru ipaPermTargetFilter: (objectclass=ipahost) ipaPermRight: write ipaPermBindRuleType: permission ipaPermissionType: V2 ipaPermissionType: MANAGED ipaPermissionType: SYSTEM cn: System: Manage Host Keytab objectClass: ipapermission objectClass: top objectClass: groupofnames objectClass: ipapermissionv2 member: cn=Host Enrollment,cn=privileges,cn=pbac,dc
Re: [Freeipa-users] replication conflicts
On 06/17/2015 11:22 AM, Alexander Frolushkin wrote: This was a usual ipa-replica-install --setup-ca --setup-dns and after that ipa-adtrust-install. No DEL found: # grep cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru ./access [17/Jun/2015:10:08:01 +0600] conn=2 op=89 SRCH base=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru scope=0 filter=(objectClass=*) attrs=ipaPermRight ipaPermTargetFilter ipaPermBindRuleType ipaPermissionType cn objectClass memberOf member ipaPermTarget ipaPermDefaultAttr ipaPermLocation ipaPermIncludedAttr ipaPermExcludedAttr [17/Jun/2015:10:08:01 +0600] conn=2 op=91 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru There is something I miss. conn=2 op=91 was a direct update on replica26 (not replicated) because it received its own CSN=5580f321001a. But it created a conflict entry, so at that time it existed the same entry (the one created 20150408070720Z) . So the direct update should have been rejected. Would you check if the replicaID=26 is unique in the topology (list-ruv for example) ? [17/Jun/2015:14:39:46 +0600] conn=237 op=93 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru It is also possible this entry on affected servers was previously duplicated and not correctly managed to delete (more recent dup was deleted). Is there any natural way to fix such issues? Maybe ipa-replica-manage force-sync, or ipa-replica-manage re-initialize on affected site servers from normal servers could help? WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 *From:*thierry bordaz [mailto:tbor...@redhat.com] *Sent:* Wednesday, June 17, 2015 3:15 PM *To:* Alexander Frolushkin (SIB) *Cc:* 'Ludwig Krispenz'; freeipa-users@redhat.com *Subject:* Re: [Freeipa-users] replication conflicts Hello Alexander, How did you initialize that new replica 26. Either 'cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru' was not part of the total init data, or a DEL of that entry happened on replica 26 (before a new ADD) but the DEL was not replicated to replica12. Would you check in replica26 access logs if that entry was deleted ? thanks theirry On 06/17/2015 11:03 AM, Alexander Frolushkin wrote: This is correct, thank you for understanding and for helping! Replica with id 26 was created today, this is our new server which was included in domain just a few hours ago. Looks like this dup came right after this new replica creation. WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 *From:*Ludwig Krispenz [mailto:lkris...@redhat.com] *Sent:* Wednesday, June 17, 2015 2:58 PM *To:* Alexander Frolushkin (SIB) *Cc:* freeipa-users@redhat.com mailto:freeipa-users@redhat.com *Subject:* Re: [Freeipa-users] replication conflicts Hi, you did send the data directly to me, maybe not wanting to share them to everyone. I'll continue discussion here, trying to be careful. The good entry was created in April on replica 12 0x0c createTimestamp;vucsn-5524d42b0067000c: 20150408070720Z the nsuniqueid entry was created today on replica 26 0x1a createTimestamp;vucsn-5580f321001a: 20150617040801Z if the original entry would have existed on replica26 the new add should have been rejected, if it was not there the question is why. Do you have any additional info on replica 26, when was it created, was it disconnected for some time ?? Ludwig On 06/17/2015 08:13 AM, Alexander Frolushkin wrote: Hello. Another example. Today appeared on servers of different site. Original LDIF: # extended LDIF # # LDAPv3 # base cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru with scope subtree # filter: (objectclass=*) # requesting: ALL # # System: Manage Host Keytab, permissions, pbac, unix.megafon.ru dn: cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc =ru ipaPermTargetFilter: (objectclass=ipahost) ipaPermRight: write ipaPermBindRuleType: permission ipaPermissionType: V2 ipaPermissionType: MANAGED ipaPermissionType: SYSTEM cn: System: Manage Host Keytab objectClass: ipapermission objectClass: top objectClass: groupofnames objectClass: ipapermissionv2 member: cn=Host Enrollment,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru member: cn=Host Administrators,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru ipaPermDefaultAttr: krbprincipalkey ipaPermDefaultAttr: krblastpwdchange ipaPermLocation: cn=computers,cn=accounts,dc=unix,dc=megafon,dc=ru
Re: [Freeipa-users] replication conflicts
Unfortunately, number of duplicates grows dramatically on most sites. Some servers already have over 40 duplicates. Could you please say, may I use re-initialize on falling replica from the good one to fix this? WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: Ludwig Krispenz [mailto:lkris...@redhat.com] Sent: Wednesday, June 17, 2015 4:35 PM To: Alexander Frolushkin (SIB) Cc: 'thierry bordaz'; freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts conn=237 is from 10.99.75.82 which replica is this ? On 06/17/2015 12:13 PM, Alexander Frolushkin wrote: This is not a good news, because replica id 20 is not exist for a some days already. It was recreated and now have id 23 WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: thierry bordaz [mailto:tbor...@redhat.com] Sent: Wednesday, June 17, 2015 4:10 PM To: Alexander Frolushkin (SIB) Cc: 'Ludwig Krispenz'; freeipa-users@redhat.commailto:freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts On 06/17/2015 11:56 AM, Alexander Frolushkin wrote: Will this be enough? # grep conn=237 op=93 ./access [17/Jun/2015:14:39:46 +0600] conn=237 op=93 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru [17/Jun/2015:14:39:46 +0600] conn=237 op=93 RESULT err=0 tag=105 nentries=0 etime=0 csn=555ac9360014 This operation is a replicated one and the CSN is from May 19th. So why a replica (26) created today was initialized without that entry ? This updates was originated from replica20. Was it stopped and restarted recently ? # grep conn=293 ./access [17/Jun/2015:15:33:04 +0600] conn=293 fd=75 slot=75 connection from 10.99.75.82 to 10.61.8.2 [17/Jun/2015:15:33:04 +0600] conn=293 op=0 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:15:33:04 +0600] conn=293 op=0 RESULT err=14 tag=97 nentries=0 etime=0, SASL bind in progress [17/Jun/2015:15:33:04 +0600] conn=293 op=1 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:15:33:04 +0600] conn=293 op=1 RESULT err=14 tag=97 nentries=0 etime=0, SASL bind in progress [17/Jun/2015:15:33:04 +0600] conn=293 op=2 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:15:33:04 +0600] conn=293 op=2 RESULT err=0 tag=97 nentries=0 etime=0 dn=krbprincipalname=ldap/msk-rhidm-03.unix.megafon...@unix.megafon.ru,cn=services,cn=accounts,dc=unix,dc=megafon,dc=rumailto:krbprincipalname=ldap/msk-rhidm-03.unix.megafon...@unix.megafon.ru,cn=services,cn=accounts,dc=unix,dc=megafon,dc=ru WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: Ludwig Krispenz [mailto:lkris...@redhat.com] Sent: Wednesday, June 17, 2015 3:53 PM To: thierry bordaz Cc: Alexander Frolushkin (SIB); freeipa-users@redhat.commailto:freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts On 06/17/2015 11:45 AM, thierry bordaz wrote: On 06/17/2015 11:22 AM, Alexander Frolushkin wrote: This was a usual ipa-replica-install --setup-ca --setup-dns and after that ipa-adtrust-install. No DEL found: # grep cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru ./access [17/Jun/2015:10:08:01 +0600] conn=2 op=89 SRCH base=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru scope=0 filter=(objectClass=*) attrs=ipaPermRight ipaPermTargetFilter ipaPermBindRuleType ipaPermissionType cn objectClass memberOf member ipaPermTarget ipaPermDefaultAttr ipaPermLocation ipaPermIncludedAttr ipaPermExcludedAttr [17/Jun/2015:10:08:01 +0600] conn=2 op=91 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru There is something I miss. conn=2 op=91 was a direct update on replica26 (not replicated) because it received its own CSN=5580f321001a. But it created a conflict entry, so at that time it existed the same entry (the one created 20150408070720Z) . So the direct update should have been rejected. I think the search in op=89 did not return an entry, so it was added in op 91, that seems to be ok, but then 4 hrs later there is conn=237 adding it again. Alexander, could you get the complete 'conn=237 op=93' and also the start of conn 293, to show where teh connection comes from Would you check if the replicaID=26 is unique in the topology (list-ruv for example) ? [17/Jun/2015:14:39:46 +0600] conn=237 op=93 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru It is also possible this entry on affected servers was previously duplicated and not correctly managed to delete (more recent dup was deleted). Is there any natural way to fix such issues? Maybe ipa-replica-manage force-sync, or ipa-replica-manage re-initialize on affected site servers from normal servers could help? WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: thierry bordaz [mailto:tbor...@redhat.com] Sent: Wednesday, June 17, 2015 3:15 PM To: Alexander Frolushkin (SIB) Cc: 'Ludwig Krispenz
Re: [Freeipa-users] replication conflicts
Hello Alexander, How did you initialize that new replica 26. Either 'cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru' was not part of the total init data, or a DEL of that entry happened on replica 26 (before a new ADD) but the DEL was not replicated to replica12. Would you check in replica26 access logs if that entry was deleted ? thanks theirry On 06/17/2015 11:03 AM, Alexander Frolushkin wrote: This is correct, thank you for understanding and for helping! Replica with id 26 was created today, this is our new server which was included in domain just a few hours ago. Looks like this dup came right after this new replica creation. WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 *From:*Ludwig Krispenz [mailto:lkris...@redhat.com] *Sent:* Wednesday, June 17, 2015 2:58 PM *To:* Alexander Frolushkin (SIB) *Cc:* freeipa-users@redhat.com *Subject:* Re: [Freeipa-users] replication conflicts Hi, you did send the data directly to me, maybe not wanting to share them to everyone. I'll continue discussion here, trying to be careful. The good entry was created in April on replica 12 0x0c createTimestamp;vucsn-5524d42b0067000c: 20150408070720Z the nsuniqueid entry was created today on replica 26 0x1a createTimestamp;vucsn-5580f321001a: 20150617040801Z if the original entry would have existed on replica26 the new add should have been rejected, if it was not there the question is why. Do you have any additional info on replica 26, when was it created, was it disconnected for some time ?? Ludwig On 06/17/2015 08:13 AM, Alexander Frolushkin wrote: Hello. Another example. Today appeared on servers of different site. Original LDIF: # extended LDIF # # LDAPv3 # base cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru with scope subtree # filter: (objectclass=*) # requesting: ALL # # System: Manage Host Keytab, permissions, pbac, unix.megafon.ru dn: cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc =ru ipaPermTargetFilter: (objectclass=ipahost) ipaPermRight: write ipaPermBindRuleType: permission ipaPermissionType: V2 ipaPermissionType: MANAGED ipaPermissionType: SYSTEM cn: System: Manage Host Keytab objectClass: ipapermission objectClass: top objectClass: groupofnames objectClass: ipapermissionv2 member: cn=Host Enrollment,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru member: cn=Host Administrators,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru ipaPermDefaultAttr: krbprincipalkey ipaPermDefaultAttr: krblastpwdchange ipaPermLocation: cn=computers,cn=accounts,dc=unix,dc=megafon,dc=ru # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 Duplicate: # extended LDIF # # LDAPv3 # base cn=System: Manage Host Keytab+nsuniqueid=708bba65-14a611e5-8a48fd19-df27ff01,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru with scope subtree # filter: (objectclass=*) # requesting: ALL # # System: Manage Host Keytab + 708bba65-14a611e5-8a48fd19-df27ff01, permissio ns, pbac, unix.megafon.ru dn: cn=System: Manage Host Keytab+nsuniqueid=708bba65-14a611e5-8a48fd19-df27ff 01,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru ipaPermTargetFilter: (objectclass=ipahost) ipaPermRight: write ipaPermBindRuleType: permission ipaPermissionType: V2 ipaPermissionType: MANAGED ipaPermissionType: SYSTEM cn: System: Manage Host Keytab objectClass: ipapermission objectClass: top objectClass: groupofnames objectClass: ipapermissionv2 member: cn=Host Enrollment,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru member: cn=Host Administrators,cn=privileges,cn=pbac,dc=unix,dc=megafon,dc=ru ipaPermDefaultAttr: krbprincipalkey ipaPermDefaultAttr: krblastpwdchange ipaPermLocation: cn=computers,cn=accounts,dc=unix,dc=megafon,dc=ru # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 No other servers in IPA domain have such duplicates. WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 *From:*freeipa-users-boun...@redhat.com mailto:freeipa-users-boun...@redhat.com [mailto:freeipa-users-boun...@redhat.com] *On Behalf Of *Ludwig Krispenz *Sent:* Tuesday, June 16, 2015 3:52 PM *To:* freeipa-users@redhat.com mailto:freeipa-users@redhat.com *Subject:* Re: [Freeipa-users] replication conflicts On 06/16/2015 11:42 AM, Alexander Frolushkin wrote: Hello. Just to remind if somebody still not familiar with our IPA installation J We currently have 18 IPA servers in domain, on 8 sites in different regions across the Russia
Re: [Freeipa-users] replication conflicts
On 06/17/2015 11:52 AM, Ludwig Krispenz wrote: On 06/17/2015 11:45 AM, thierry bordaz wrote: On 06/17/2015 11:22 AM, Alexander Frolushkin wrote: This was a usual ipa-replica-install --setup-ca --setup-dns and after that ipa-adtrust-install. No DEL found: # grep cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru ./access [17/Jun/2015:10:08:01 +0600] conn=2 op=89 SRCH base=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru scope=0 filter=(objectClass=*) attrs=ipaPermRight ipaPermTargetFilter ipaPermBindRuleType ipaPermissionType cn objectClass memberOf member ipaPermTarget ipaPermDefaultAttr ipaPermLocation ipaPermIncludedAttr ipaPermExcludedAttr [17/Jun/2015:10:08:01 +0600] conn=2 op=91 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru There is something I miss. conn=2 op=91 was a direct update on replica26 (not replicated) because it received its own CSN=5580f321001a. But it created a conflict entry, so at that time it existed the same entry (the one created 20150408070720Z) . So the direct update should have been rejected. I think the search in op=89 did not return an entry, so it was added in op 91, that seems to be ok, but then 4 hrs later there is conn=237 adding it again. Alexander, could you get the complete 'conn=237 op=93' and also the start of conn 293, to show where teh connection comes from of course conn=237 Would you check if the replicaID=26 is unique in the topology (list-ruv for example) ? [17/Jun/2015:14:39:46 +0600] conn=237 op=93 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru It is also possible this entry on affected servers was previously duplicated and not correctly managed to delete (more recent dup was deleted). Is there any natural way to fix such issues? Maybe ipa-replica-manage force-sync, or ipa-replica-manage re-initialize on affected site servers from normal servers could help? WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 *From:*thierry bordaz [mailto:tbor...@redhat.com] *Sent:* Wednesday, June 17, 2015 3:15 PM *To:* Alexander Frolushkin (SIB) *Cc:* 'Ludwig Krispenz'; freeipa-users@redhat.com *Subject:* Re: [Freeipa-users] replication conflicts Hello Alexander, How did you initialize that new replica 26. Either 'cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru' was not part of the total init data, or a DEL of that entry happened on replica 26 (before a new ADD) but the DEL was not replicated to replica12. Would you check in replica26 access logs if that entry was deleted ? thanks theirry On 06/17/2015 11:03 AM, Alexander Frolushkin wrote: This is correct, thank you for understanding and for helping! Replica with id 26 was created today, this is our new server which was included in domain just a few hours ago. Looks like this dup came right after this new replica creation. WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 *From:*Ludwig Krispenz [mailto:lkris...@redhat.com] *Sent:* Wednesday, June 17, 2015 2:58 PM *To:* Alexander Frolushkin (SIB) *Cc:* freeipa-users@redhat.com mailto:freeipa-users@redhat.com *Subject:* Re: [Freeipa-users] replication conflicts Hi, you did send the data directly to me, maybe not wanting to share them to everyone. I'll continue discussion here, trying to be careful. The good entry was created in April on replica 12 0x0c createTimestamp;vucsn-5524d42b0067000c: 20150408070720Z the nsuniqueid entry was created today on replica 26 0x1a createTimestamp;vucsn-5580f321001a: 20150617040801Z if the original entry would have existed on replica26 the new add should have been rejected, if it was not there the question is why. Do you have any additional info on replica 26, when was it created, was it disconnected for some time ?? Ludwig On 06/17/2015 08:13 AM, Alexander Frolushkin wrote: Hello. Another example. Today appeared on servers of different site. Original LDIF: # extended LDIF # # LDAPv3 # base cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru with scope subtree # filter: (objectclass=*) # requesting: ALL # # System: Manage Host Keytab, permissions, pbac, unix.megafon.ru dn: cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc =ru ipaPermTargetFilter: (objectclass=ipahost) ipaPermRight: write ipaPermBindRuleType: permission ipaPermissionType: V2 ipaPermissionType: MANAGED ipaPermissionType: SYSTEM cn: System: Manage Host Keytab objectClass: ipapermission objectClass: top objectClass
Re: [Freeipa-users] replication conflicts
On 06/17/2015 11:45 AM, thierry bordaz wrote: On 06/17/2015 11:22 AM, Alexander Frolushkin wrote: This was a usual ipa-replica-install --setup-ca --setup-dns and after that ipa-adtrust-install. No DEL found: # grep cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru ./access [17/Jun/2015:10:08:01 +0600] conn=2 op=89 SRCH base=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru scope=0 filter=(objectClass=*) attrs=ipaPermRight ipaPermTargetFilter ipaPermBindRuleType ipaPermissionType cn objectClass memberOf member ipaPermTarget ipaPermDefaultAttr ipaPermLocation ipaPermIncludedAttr ipaPermExcludedAttr [17/Jun/2015:10:08:01 +0600] conn=2 op=91 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru There is something I miss. conn=2 op=91 was a direct update on replica26 (not replicated) because it received its own CSN=5580f321001a. But it created a conflict entry, so at that time it existed the same entry (the one created 20150408070720Z) . So the direct update should have been rejected. I think the search in op=89 did not return an entry, so it was added in op 91, that seems to be ok, but then 4 hrs later there is conn=237 adding it again. Alexander, could you get the complete 'conn=237 op=93' and also the start of conn 293, to show where teh connection comes from Would you check if the replicaID=26 is unique in the topology (list-ruv for example) ? [17/Jun/2015:14:39:46 +0600] conn=237 op=93 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru It is also possible this entry on affected servers was previously duplicated and not correctly managed to delete (more recent dup was deleted). Is there any natural way to fix such issues? Maybe ipa-replica-manage force-sync, or ipa-replica-manage re-initialize on affected site servers from normal servers could help? WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 *From:*thierry bordaz [mailto:tbor...@redhat.com] *Sent:* Wednesday, June 17, 2015 3:15 PM *To:* Alexander Frolushkin (SIB) *Cc:* 'Ludwig Krispenz'; freeipa-users@redhat.com *Subject:* Re: [Freeipa-users] replication conflicts Hello Alexander, How did you initialize that new replica 26. Either 'cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru' was not part of the total init data, or a DEL of that entry happened on replica 26 (before a new ADD) but the DEL was not replicated to replica12. Would you check in replica26 access logs if that entry was deleted ? thanks theirry On 06/17/2015 11:03 AM, Alexander Frolushkin wrote: This is correct, thank you for understanding and for helping! Replica with id 26 was created today, this is our new server which was included in domain just a few hours ago. Looks like this dup came right after this new replica creation. WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 *From:*Ludwig Krispenz [mailto:lkris...@redhat.com] *Sent:* Wednesday, June 17, 2015 2:58 PM *To:* Alexander Frolushkin (SIB) *Cc:* freeipa-users@redhat.com mailto:freeipa-users@redhat.com *Subject:* Re: [Freeipa-users] replication conflicts Hi, you did send the data directly to me, maybe not wanting to share them to everyone. I'll continue discussion here, trying to be careful. The good entry was created in April on replica 12 0x0c createTimestamp;vucsn-5524d42b0067000c: 20150408070720Z the nsuniqueid entry was created today on replica 26 0x1a createTimestamp;vucsn-5580f321001a: 20150617040801Z if the original entry would have existed on replica26 the new add should have been rejected, if it was not there the question is why. Do you have any additional info on replica 26, when was it created, was it disconnected for some time ?? Ludwig On 06/17/2015 08:13 AM, Alexander Frolushkin wrote: Hello. Another example. Today appeared on servers of different site. Original LDIF: # extended LDIF # # LDAPv3 # base cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru with scope subtree # filter: (objectclass=*) # requesting: ALL # # System: Manage Host Keytab, permissions, pbac, unix.megafon.ru dn: cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc =ru ipaPermTargetFilter: (objectclass=ipahost) ipaPermRight: write ipaPermBindRuleType: permission ipaPermissionType: V2 ipaPermissionType: MANAGED ipaPermissionType: SYSTEM cn: System: Manage Host Keytab objectClass: ipapermission objectClass: top objectClass: groupofnames objectClass: ipapermissionv2 member: cn
Re: [Freeipa-users] replication conflicts
On 06/17/2015 11:56 AM, Alexander Frolushkin wrote: Will this be enough? # grep conn=237 op=93 ./access [17/Jun/2015:14:39:46 +0600] conn=237 op=93 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru [17/Jun/2015:14:39:46 +0600] conn=237 op=93 RESULT err=0 tag=105 nentries=0 etime=0 csn=555ac9360014 This operation is a replicated one and the CSN is from May 19th. So why a replica (26) created today was initialized without that entry ? This updates was originated from replica20. Was it stopped and restarted recently ? # grep conn=293 ./access [17/Jun/2015:15:33:04 +0600] conn=293 fd=75 slot=75 connection from 10.99.75.82 to 10.61.8.2 [17/Jun/2015:15:33:04 +0600] conn=293 op=0 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:15:33:04 +0600] conn=293 op=0 RESULT err=14 tag=97 nentries=0 etime=0, SASL bind in progress [17/Jun/2015:15:33:04 +0600] conn=293 op=1 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:15:33:04 +0600] conn=293 op=1 RESULT err=14 tag=97 nentries=0 etime=0, SASL bind in progress [17/Jun/2015:15:33:04 +0600] conn=293 op=2 BIND dn= method=sasl version=3 mech=GSSAPI [17/Jun/2015:15:33:04 +0600] conn=293 op=2 RESULT err=0 tag=97 nentries=0 etime=0 dn=krbprincipalname=ldap/msk-rhidm-03.unix.megafon...@unix.megafon.ru,cn=services,cn=accounts,dc=unix,dc=megafon,dc=ru WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 *From:*Ludwig Krispenz [mailto:lkris...@redhat.com] *Sent:* Wednesday, June 17, 2015 3:53 PM *To:* thierry bordaz *Cc:* Alexander Frolushkin (SIB); freeipa-users@redhat.com *Subject:* Re: [Freeipa-users] replication conflicts On 06/17/2015 11:45 AM, thierry bordaz wrote: On 06/17/2015 11:22 AM, Alexander Frolushkin wrote: This was a usual ipa-replica-install --setup-ca --setup-dns and after that ipa-adtrust-install. No DEL found: # grep cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru ./access [17/Jun/2015:10:08:01 +0600] conn=2 op=89 SRCH base=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru scope=0 filter=(objectClass=*) attrs=ipaPermRight ipaPermTargetFilter ipaPermBindRuleType ipaPermissionType cn objectClass memberOf member ipaPermTarget ipaPermDefaultAttr ipaPermLocation ipaPermIncludedAttr ipaPermExcludedAttr [17/Jun/2015:10:08:01 +0600] conn=2 op=91 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru There is something I miss. conn=2 op=91 was a direct update on replica26 (not replicated) because it received its own CSN=5580f321001a. But it created a conflict entry, so at that time it existed the same entry (the one created 20150408070720Z) . So the direct update should have been rejected. I think the search in op=89 did not return an entry, so it was added in op 91, that seems to be ok, but then 4 hrs later there is conn=237 adding it again. Alexander, could you get the complete 'conn=237 op=93' and also the start of conn 293, to show where teh connection comes from Would you check if the replicaID=26 is unique in the topology (list-ruv for example) ? [17/Jun/2015:14:39:46 +0600] conn=237 op=93 ADD dn=cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru It is also possible this entry on affected servers was previously duplicated and not correctly managed to delete (more recent dup was deleted). Is there any natural way to fix such issues? Maybe ipa-replica-manage force-sync, or ipa-replica-manage re-initialize on affected site servers from normal servers could help? WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 *From:*thierry bordaz [mailto:tbor...@redhat.com] *Sent:* Wednesday, June 17, 2015 3:15 PM *To:* Alexander Frolushkin (SIB) *Cc:* 'Ludwig Krispenz'; freeipa-users@redhat.com mailto:freeipa-users@redhat.com *Subject:* Re: [Freeipa-users] replication conflicts Hello Alexander, How did you initialize that new replica 26. Either 'cn=System: Manage Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru' was not part of the total init data, or a DEL of that entry happened on replica 26 (before a new ADD) but the DEL was not replicated to replica12. Would you check in replica26 access logs if that entry was deleted ? thanks theirry On 06/17/2015 11:03 AM, Alexander Frolushkin wrote: This is correct, thank you for understanding and for helping! Replica with id 26 was created today, this is our new server which was included in domain just a few hours ago. Looks like this dup came right after this new replica creation. WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 *From:*Ludwig Krispenz [mailto:lkris...@redhat.com] *Sent:* Wednesday, June 17, 2015 2:58 PM *To:* Alexander
Re: [Freeipa-users] replication conflicts
On 06/17/2015 02:27 PM, Alexander Frolushkin wrote:
Except:
unable to decode: {replica 22} 5576b83e00020016 5576ba4b00020016
unable to decode: {replica 20} 55716e5700030014 55716e5700030014
unable to decode: {replica 16} 548a81260010 548a81260010
unable to decode: {replica 24} 557fb7d400040018 557fb9a100100018
unable to decode: {replica 21} 5576ac960015 5576b52e00020015
records, all replicas are unique and have corresponding id's.
Total number of not unable to decode servers is equal to real number
of replicas in domain.
I can confirm some of cleanallruv processes was not finished correctly
after some replica remove.
Slapd on replica id 10 last restarted yesterday 15:05 server local time.
It was restarted 16/Jun/2015:15:05 and send the update one day after !
[17/Jun/2015:14:39:46 +0600] conn=237 op=93 ADD dn=cn=System: Manage
Host Keytab,cn=permissions,cn=pbac,dc=unix,dc=megafon,dc=ru
[17/Jun/2015:14:39:46 +0600] conn=237 op=93 RESULT err=0 tag=105
nentries=0 etime=0 csn=555ac9360014
May be it was very very slow.
The same question, may it help if I tomorrow will do re-initialize all
replicas from our relatively good-conditioned site?
If you reinit all replicas, it will clear their CL so such very old
updates will no long exist.
Now if it exists very slow replica (like replica10) that holds their
updates for long before replicating then it increases the possibility of
creating conflicts.
WBR,
Alexander Frolushkin
Cell +79232508764
Work +79232507764
*From:*thierry bordaz [mailto:tbor...@redhat.com]
*Sent:* Wednesday, June 17, 2015 6:16 PM
*To:* Alexander Frolushkin (SIB)
*Cc:* 'Ludwig Krispenz'; freeipa-users@redhat.com
*Subject:* Re: [Freeipa-users] replication conflicts
On 06/17/2015 01:38 PM, Alexander Frolushkin wrote:
Ok, I'll try this soon, thank you!
Also, please note, most of today dups appeared when 4 of 19
servers was very busy in IO (all our servers are VMs), because
dirsrv debug was enabled to gather logs for our case about
attrlist_replace - attr_replace (nsslapd-referral,
ldap://xxx-rhidm0x.unix.megafon.ru:389/o%3Dipaca) failed.
This message comes if you have duplicated ReplicaID. 'list-ruv' would
show you a same url appearing several times with different RID.
To be back on the original problem (dup), I wonder if it is not
related to cleanruv/corrupted RUV.
A new replica26 is created and is initialized from a master and it
does not contain entry 555ac9360014. That allows the creation of
5580f321001a.
Then replica10 sends 555ac9360014 to replica26. That means
that changelog/ruv of replica10 contained that update, so either
cleanallruv20 did not happen on replica10 or fail to clean its CL.
Replica10 has a very old update that was not known from the master
used to init replica26.
Was replica10 restarted recently.
thanks
theirry
errors
Also during collection some of dirsrv instances hangs and was restarted.
WBR,
Alexander Frolushkin
Cell +79232508764
Work +79232507764
*From:*Ludwig Krispenz [mailto:lkris...@redhat.com]
*Sent:* Wednesday, June 17, 2015 5:34 PM
*To:* Alexander Frolushkin (SIB)
*Cc:* 'thierry bordaz'; freeipa-users@redhat.com
mailto:freeipa-users@redhat.com
*Subject:* Re: [Freeipa-users] replication conflicts
On 06/17/2015 12:57 PM, Alexander Frolushkin wrote:
Unfortunately, number of duplicates grows dramatically on most
sites. Some servers already have over 40 duplicates.
Could you please say, may I use re-initialize on falling replica
from the good one to fix this?
If you have a good one, this should work, dups are only created when
a replicated ADD is received for an existing entry.
But what really puzzles me is that you do not have them on all
servers, something weird seems to happen, this entry seems to exist
whit several replicaids, and why would replica 10 replicate this 4 hrs
after the replica installation.
WBR,
Alexander Frolushkin
Cell +79232508764
Work +79232507764
*From:*Ludwig Krispenz [mailto:lkris...@redhat.com]
*Sent:* Wednesday, June 17, 2015 4:35 PM
*To:* Alexander Frolushkin (SIB)
*Cc:* 'thierry bordaz'; freeipa-users@redhat.com
mailto:freeipa-users@redhat.com
*Subject:* Re: [Freeipa-users] replication conflicts
conn=237 is from 10.99.75.82 which replica is this ?
On 06/17/2015 12:13 PM, Alexander Frolushkin wrote:
This is not a good news, because replica id 20 is not exist for a
some days already. It was recreated and now have id 23
WBR,
Alexander Frolushkin
Cell +79232508764
Work +79232507764
*From:*thierry bordaz [mailto:tbor...@redhat.com]
*Sent:* Wednesday, June 17, 2015 4:10 PM
*To:* Alexander Frolushkin (SIB)
*Cc:* 'Ludwig Krispenz'; freeipa-users@redhat.com
mailto:freeipa-users@redhat.com
*Subject:* Re: [Freeipa-users] replication conflicts
On 06/17/2015 11:56 AM, Alexander Frolushkin
Re: [Freeipa-users] replication conflicts
One example of duplicate: krbprincipalname=HTTP/nw-rhidm02.unix.megafon...@unix.megafon.ru+nsuniqueid=5a726d95-0e9611e5-8418a085-d3870578,cn=services,cn=accounts,dc=unix,dc=megafon,dc=ru the original one: krbprincipalname=HTTP/nw-rhidm02.unix.megafon...@unix.megafon.ru,cn=services,cn=accounts,dc=unix,dc=megafon,dc=ru On three servers placed on one site we have such duplicates. On all other servers we have only record with normal name, with content of record, which have +nsuniqueid=5a726d95-0e9611e5-8418a085-d3870578 on affected servers. Plus we have one record with no original one, only name with +nsuniqueid, and no such record on all other servers. WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: Ludwig Krispenz [mailto:lkris...@redhat.com] Sent: Tuesday, June 16, 2015 5:30 PM To: Alexander Frolushkin (SIB) Cc: freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts On 06/16/2015 12:44 PM, Alexander Frolushkin wrote: It looks like our duplicates have some internal source, it source is not a client system, but one of our IPA servers. to get these kind of conflict two servers have to be involved if you say internal source, what kind of entries are affected ? do you mean these entries are created internally on server by a plugin ? Is it possible to get such duplicate records in combination of replication multipath and some clock skew (it is not ideally synchronized because of very big distances between sites)? the clock skew should have no effect, the replication protocol additinally manages it own time used in genratio of CSNs and tries to synchronize time, it could affect the oreder changes are applied during replication, but for these conflicts there have to be two independent ADDs WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: freeipa-users-boun...@redhat.commailto:freeipa-users-boun...@redhat.com [mailto:freeipa-users-boun...@redhat.com] On Behalf Of Ludwig Krispenz Sent: Tuesday, June 16, 2015 3:52 PM To: freeipa-users@redhat.commailto:freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts On 06/16/2015 11:42 AM, Alexander Frolushkin wrote: Hello. Just to remind if somebody still not familiar with our IPA installation :) We currently have 18 IPA servers in domain, on 8 sites in different regions across the Russia. And now, our new problem. Regularly we getting a nsds5ReplConflict records on some of our servers, very often on servers from specific site. Usually it is simply a doubles and we can remove the renamed change to get everything back. But why do we have them at all? May be someone could explain, how we can detect the cause of this replication conflicts? if you are talking about having two duplicate entries, one: uid=x,suffix one: nsuniqueid=+uid=x,suffix these entries appear if the entry uid=x was added, simultaneously, on two servers. I think this can happen if a client tries to add an entry and if it doesn't get a response in some time retries on another server. to find out which client this is you need to check on which servers the entries were originally added and then see which client was doing it Sometime it is moderately harmful, because, for example HBAC stops working on specific server while doubles still present. Thanks in forward... WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 Информация в этом сообщении предназначена исключительно для конкретных лиц, которым она адресована. В сообщении может содержаться конфиденциальная информация, которая не может быть раскрыта или использована кем-либо, кроме адресатов. Если вы не адресат этого сообщения, то использование, переадресация, копирование или распространение содержания сообщения или его части незаконно и запрещено. Если Вы получили это сообщение ошибочно, пожалуйста, незамедлительно сообщите отправителю об этом и удалите со всем содержимым само сообщение и любые возможные его копии и приложения. The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. The contents may not be disclosed or used by anyone other than the addressee. If you are not the intended recipient(s), any use, disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. If you have received this communication in error please notify us immediately by responding to this email and then delete the e-mail and all attachments and any copies thereof. (c)20mf50 Информация в этом сообщении предназначена исключительно для конкретных лиц, которым она адресована. В сообщении может содержаться конфиденциальная информация, которая не может быть раскрыта или использована кем-либо, кроме адресатов
Re: [Freeipa-users] replication conflicts
It looks like our duplicates have some internal source, it source is not a client system, but one of our IPA servers. Is it possible to get such duplicate records in combination of replication multipath and some clock skew (it is not ideally synchronized because of very big distances between sites)? WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 From: freeipa-users-boun...@redhat.com [mailto:freeipa-users-boun...@redhat.com] On Behalf Of Ludwig Krispenz Sent: Tuesday, June 16, 2015 3:52 PM To: freeipa-users@redhat.com Subject: Re: [Freeipa-users] replication conflicts On 06/16/2015 11:42 AM, Alexander Frolushkin wrote: Hello. Just to remind if somebody still not familiar with our IPA installation :) We currently have 18 IPA servers in domain, on 8 sites in different regions across the Russia. And now, our new problem. Regularly we getting a nsds5ReplConflict records on some of our servers, very often on servers from specific site. Usually it is simply a doubles and we can remove the renamed change to get everything back. But why do we have them at all? May be someone could explain, how we can detect the cause of this replication conflicts? if you are talking about having two duplicate entries, one: uid=x,suffix one: nsuniqueid=+uid=x,suffix these entries appear if the entry uid=x was added, simultaneously, on two servers. I think this can happen if a client tries to add an entry and if it doesn't get a response in some time retries on another server. to find out which client this is you need to check on which servers the entries were originally added and then see which client was doing it Sometime it is moderately harmful, because, for example HBAC stops working on specific server while doubles still present. Thanks in forward... WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 Информация в этом сообщении предназначена исключительно для конкретных лиц, которым она адресована. В сообщении может содержаться конфиденциальная информация, которая не может быть раскрыта или использована кем-либо, кроме адресатов. Если вы не адресат этого сообщения, то использование, переадресация, копирование или распространение содержания сообщения или его части незаконно и запрещено. Если Вы получили это сообщение ошибочно, пожалуйста, незамедлительно сообщите отправителю об этом и удалите со всем содержимым само сообщение и любые возможные его копии и приложения. The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. The contents may not be disclosed or used by anyone other than the addressee. If you are not the intended recipient(s), any use, disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. If you have received this communication in error please notify us immediately by responding to this email and then delete the e-mail and all attachments and any copies thereof. (c)20mf50 Информация в этом сообщении предназначена исключительно для конкретных лиц, которым она адресована. В сообщении может содержаться конфиденциальная информация, которая не может быть раскрыта или использована кем-либо, кроме адресатов. Если вы не адресат этого сообщения, то использование, переадресация, копирование или распространение содержания сообщения или его части незаконно и запрещено. Если Вы получили это сообщение ошибочно, пожалуйста, незамедлительно сообщите отправителю об этом и удалите со всем содержимым само сообщение и любые возможные его копии и приложения. The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. The contents may not be disclosed or used by anyone other than the addressee. If you are not the intended recipient(s), any use, disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. If you have received this communication in error please notify us immediately by responding to this email and then delete the e-mail and all attachments and any copies thereof. (c)20mf50 -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] replication conflicts
On 06/16/2015 11:42 AM, Alexander Frolushkin wrote: Hello. Just to remind if somebody still not familiar with our IPA installation J We currently have 18 IPA servers in domain, on 8 sites in different regions across the Russia. And now, our new problem. Regularly we getting a nsds5ReplConflict records on some of our servers, very often on servers from specific site. Usually it is simply a doubles and we can remove the renamed change to get everything back. But why do we have them at all? May be someone could explain, how we can detect the cause of this replication conflicts? if you are talking about having two duplicate entries, one: uid=x,suffix one: nsuniqueid=+uid=x,suffix these entries appear if the entry uid=x was added, simultaneously, on two servers. I think this can happen if a client tries to add an entry and if it doesn't get a response in some time retries on another server. to find out which client this is you need to check on which servers the entries were originally added and then see which client was doing it Sometime it is moderately harmful, because, for example HBAC stops working on specific server while doubles still present. Thanks in forward... WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 ?? ? ? ? ? ??? ?? ???, ??? ??? ??. ? ? ? ??? ??, ??? ?? ? ??? ???-, ? ?. ?? ?? ??? ? ?, ?? ?, ?, ??? ??? ??? ?? ? ??? ??? ? ? ? ?. ?? ??? ? , ??, ??? ??? ?? ? ??? ?? ?? ? ? ? ? ??? ? ? ??. The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. The contents may not be disclosed or used by anyone other than the addressee. If you are not the intended recipient(s), any use, disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. If you have received this communication in error please notify us immediately by responding to this email and then delete the e-mail and all attachments and any copies thereof. (c)20mf50 -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] replication conflicts
On 06/16/2015 12:44 PM, Alexander Frolushkin wrote: It looks like our duplicates have some internal source, it source is not a client system, but one of our IPA servers. to get these kind of conflict two servers have to be involved if you say internal source, what kind of entries are affected ? do you mean these entries are created internally on server by a plugin ? Is it possible to get such duplicate records in combination of replication multipath and some clock skew (it is not ideally synchronized because of very big distances between sites)? the clock skew should have no effect, the replication protocol additinally manages it own time used in genratio of CSNs and tries to synchronize time, it could affect the oreder changes are applied during replication, but for these conflicts there have to be two independent ADDs WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 *From:*freeipa-users-boun...@redhat.com [mailto:freeipa-users-boun...@redhat.com] *On Behalf Of *Ludwig Krispenz *Sent:* Tuesday, June 16, 2015 3:52 PM *To:* freeipa-users@redhat.com *Subject:* Re: [Freeipa-users] replication conflicts On 06/16/2015 11:42 AM, Alexander Frolushkin wrote: Hello. Just to remind if somebody still not familiar with our IPA installation J We currently have 18 IPA servers in domain, on 8 sites in different regions across the Russia. And now, our new problem. Regularly we getting a nsds5ReplConflict records on some of our servers, very often on servers from specific site. Usually it is simply a doubles and we can remove the renamed change to get everything back. But why do we have them at all? May be someone could explain, how we can detect the cause of this replication conflicts? if you are talking about having two duplicate entries, one: uid=x,suffix one: nsuniqueid=+uid=x,suffix these entries appear if the entry uid=x was added, simultaneously, on two servers. I think this can happen if a client tries to add an entry and if it doesn't get a response in some time retries on another server. to find out which client this is you need to check on which servers the entries were originally added and then see which client was doing it Sometime it is moderately harmful, because, for example HBAC stops working on specific server while doubles still present. Thanks in forward... WBR, Alexander Frolushkin Cell +79232508764 Work +79232507764 Информация в этом сообщении предназначена исключительно для конкретных лиц, которым она адресована. В сообщении может содержаться конфиденциальная информация, которая не может быть раскрыта или использована кем-либо, кроме адресатов. Если вы не адресат этого сообщения, то использование, переадресация, копирование или распространение содержания сообщения или его части незаконно и запрещено. Если Вы получили это сообщение ошибочно, пожалуйста, незамедлительно сообщите отправителю об этом и удалите со всем содержимым само сообщение и любые возможные его копии и приложения. The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. The contents may not be disclosed or used by anyone other than the addressee. If you are not the intended recipient(s), any use, disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. If you have received this communication in error please notify us immediately by responding to this email and then delete the e-mail and all attachments and any copies thereof. (c)20mf50 Информация в этом сообщении предназначена исключительно для конкретных лиц, которым она адресована. В сообщении может содержаться конфиденциальная информация, которая не может быть раскрыта или использована кем-либо, кроме адресатов. Если вы не адресат этого сообщения, то использование, переадресация, копирование или распространение содержания сообщения или его части незаконно и запрещено. Если Вы получили это сообщение ошибочно, пожалуйста, незамедлительно сообщите отправителю об этом и удалите со всем содержимым само сообщение и любые возможные его копии и приложения. The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. The contents may not be disclosed or used by anyone other than the addressee. If you are not the intended recipient(s), any use, disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it is prohibited and may
