Re: [Freeipa-users] Server replication stopped working

2016-09-27 Thread Ludwig Krispenz


On 09/27/2016 06:04 PM, Youenn PIOLET wrote:

Hi Ludwig,

Version:
389-ds-base-1.3.4.0-33.el7_2.x86_64
we have identified an issue with this version, it includes a fix for 
389-ds ticket #48766, which was incomplete and resolved shortly after 
the release of this version (it is missing the latest patch for #49766 
and for #48954).
You can try to go back to 1.3.4.0-32 or if you have support get a hotfix 
from our support.


Sorry for this,
Ludwig


The timestamp probably matches the last time I've done a 
ipa-replica-manage re-initialize.
I have to do it every day (many times a day actually!), as replication 
is broken, This CSN changes all the time.


My main goal is to rebuilt everything from a clean base.
I've got no master without errors.

What is the easiest way to rebuilt everything?
ipa-[cs]replica-manage re-initialize isn't very effective.

Thanks by advance,
Regards

--
Youenn Piolet
piole...@gmail.com 
/
/

2016-09-26 9:42 GMT+02:00 Ludwig Krispenz >:



On 09/25/2016 09:35 PM, Youenn PIOLET wrote:

Hi there,

Same issue for me in a my 15 ipa-servers multi-master grid just
after the update.
The replication is completely broken except on 3/15 nodes.

This is the second time I have to fully reinitialize the whole
cluster for similar reason. I don't know what to do to clean this
mess...
For more information: this cluster has been initialized on a
fedora 4.1.4 more than one year ago then complemetely migrated to
Centos 7, IPA 4.2.

what is the exact version of 389-ds-base you are running ?

did these errors come out of the blue or are they related to some
activities ? The csn which is not found has a timestamp of "Thu,
22 Sep 2016 15:59:08 GMT" did anything happen around this time ?



Example on fr-master03 error logs:

[25/Sep/2016:19:27:31 +] NSMMReplicationPlugin - changelog
program - agmt="cn=meTofr-master01.domain" (fr-master01:389): CSN
57e3ffcc0003001a not found, we aren't as up to date, or we purged
[25/Sep/2016:19:27:31 +] NSMMReplicationPlugin -
agmt="cn=meTofr-master01.domain" (fr-master01:389): Data required
to update replica has been purged. The replica must be reinitialized.
[25/Sep/2016:19:27:31 +] NSMMReplicationPlugin -
agmt="cn=meTofr-master01.domain" (fr-master01:389): Incremental
update failed and requires administrator action
ipa: INFO: The ipactl command was successful
[25/Sep/2016:19:27:35 +] agmt="cn=meTofr-master02.domain"
(fr-master02:389) - Can't locate CSN 57e3ffcc0003001a in the
changelog (DB rc=-30988). If replication stops, the consumer may
need to be reinitialized.
[25/Sep/2016:19:27:35 +] NSMMReplicationPlugin - changelog
program - agmt="cn=meTofr-master02.domain" (fr-master02:389): CSN
57e3ffcc0003001a not found, we aren't as up to date, or we purged
[25/Sep/2016:19:27:35 +] NSMMReplicationPlugin -
agmt="cn=meTofr-master02.domain" (fr-master02:389): Data required
to update replica has been purged. The replica must be reinitialized.
[25/Sep/2016:19:27:35 +] NSMMReplicationPlugin -
agmt="cn=meTofr-master02.domain" (fr-master02:389): Incremental
update failed and requires administrator action

Regards,

--
Youenn Piolet
piole...@gmail.com 
/
/

2016-09-23 17:51 GMT+02:00 Mike Driscoll
mailto:mike.drisc...@oracle.com>>:

Hello.  I have four IPA servers replicating in full mesh. 
All four servers are running

ipa-server-4.2.0-15.0.1.el7_2.19.x86_64.

This was working for some time but now I see that no
replication is occurring automatically at present.

When I update a user attribute on an IPA server, I see errors
like these:
[22/Sep/2016:16:53:49 -0700] attrlist_replace - attr_replace
(nsslapd-referral, ldap://ldap03.xx.com:389/o%3Dipaca) failed.
[22/Sep/2016:16:58:56 -0700] NSMMReplicationPlugin -
agmt="cn=masterAgreement1-ldap03.xx.com
-pki-tomcat" (ldap03:389):
Incremental update failed and requires administrator action

I can reinitialize without errors.
ipa-csreplica-manage re-initialize --from=ldap01.xx.com

ipa-replica-manage re-initialize --from=ldap01.xx.com

Afterwards I see my attribute (and other) changes are
replicated on each server I re-initialize from.  But
subsequently, replication doesn’t seem to be happening.

I reinitialized according to the steps in Table 8.7,
“Replication Errors”, but subsequent replication isn’t
occurring. Any suggestions?  Is it safe to identify one of my
four servers as containing up-to-date data, then sever and
reinstate replicati

Re: [Freeipa-users] Server replication stopped working

2016-09-27 Thread Youenn PIOLET
Hi Ludwig,

Version:
389-ds-base-1.3.4.0-33.el7_2.x86_64

The timestamp probably matches the last time I've done a ipa-replica-manage
re-initialize.
I have to do it every day (many times a day actually!), as replication is
broken, This CSN changes all the time.

My main goal is to rebuilt everything from a clean base.
I've got no master without errors.

What is the easiest way to rebuilt everything?
ipa-[cs]replica-manage re-initialize isn't very effective.

Thanks by advance,
Regards

--
Youenn Piolet
piole...@gmail.com


2016-09-26 9:42 GMT+02:00 Ludwig Krispenz :

>
> On 09/25/2016 09:35 PM, Youenn PIOLET wrote:
>
> Hi there,
>
> Same issue for me in a my 15 ipa-servers multi-master grid just after the
> update.
> The replication is completely broken except on 3/15 nodes.
>
> This is the second time I have to fully reinitialize the whole cluster for
> similar reason. I don't know what to do to clean this mess...
> For more information: this cluster has been initialized on a fedora 4.1.4
> more than one year ago then complemetely migrated to Centos 7, IPA 4.2.
>
> what is the exact version of 389-ds-base you are running ?
>
> did these errors come out of the blue or are they related to some
> activities ? The csn which is not found has a timestamp of "Thu, 22 Sep
> 2016 15:59:08 GMT" did anything happen around this time ?
>
>
> Example on fr-master03 error logs:
>
> [25/Sep/2016:19:27:31 +] NSMMReplicationPlugin - changelog program -
> agmt="cn=meTofr-master01.domain" (fr-master01:389): CSN
> 57e3ffcc0003001a not found, we aren't as up to date, or we purged
> [25/Sep/2016:19:27:31 +] NSMMReplicationPlugin -
> agmt="cn=meTofr-master01.domain" (fr-master01:389): Data required to
> update replica has been purged. The replica must be reinitialized.
> [25/Sep/2016:19:27:31 +] NSMMReplicationPlugin -
> agmt="cn=meTofr-master01.domain" (fr-master01:389): Incremental update
> failed and requires administrator action
> ipa: INFO: The ipactl command was successful
> [25/Sep/2016:19:27:35 +] agmt="cn=meTofr-master02.domain"
> (fr-master02:389) - Can't locate CSN 57e3ffcc0003001a in the changelog
> (DB rc=-30988). If replication stops, the consumer may need to be
> reinitialized.
> [25/Sep/2016:19:27:35 +] NSMMReplicationPlugin - changelog program -
> agmt="cn=meTofr-master02.domain" (fr-master02:389): CSN
> 57e3ffcc0003001a not found, we aren't as up to date, or we purged
> [25/Sep/2016:19:27:35 +] NSMMReplicationPlugin -
> agmt="cn=meTofr-master02.domain" (fr-master02:389): Data required to
> update replica has been purged. The replica must be reinitialized.
> [25/Sep/2016:19:27:35 +] NSMMReplicationPlugin -
> agmt="cn=meTofr-master02.domain" (fr-master02:389): Incremental update
> failed and requires administrator action
>
> Regards,
>
> --
> Youenn Piolet
> piole...@gmail.com
>
>
> 2016-09-23 17:51 GMT+02:00 Mike Driscoll :
>
>> Hello.  I have four IPA servers replicating in full mesh.  All four
>> servers are running ipa-server-4.2.0-15.0.1.el7_2.19.x86_64.
>>
>> This was working for some time but now I see that no replication is
>> occurring automatically at present.
>>
>> When I update a user attribute on an IPA server, I see errors like these:
>> [22/Sep/2016:16:53:49 -0700] attrlist_replace - attr_replace
>> (nsslapd-referral, ldap://ldap03.xx.com:389/o%3Dipaca) failed.
>> [22/Sep/2016:16:58:56 -0700] NSMMReplicationPlugin - agmt="cn=
>> masterAgreement1-ldap03.xx.com 
>> -pki-tomcat" (ldap03:389): Incremental update failed and requires
>> administrator action
>>
>> I can reinitialize without errors.
>> ipa-csreplica-manage re-initialize --from=ldap01.xx.com
>> 
>> ipa-replica-manage re-initialize --from=ldap01.xx.com
>> Afterwards I see my attribute (and other) changes are replicated on each
>> server I re-initialize from.  But subsequently, replication doesn’t seem to
>> be happening.
>>
>> I reinitialized according to the steps in Table 8.7, “Replication
>> Errors”, but subsequent replication isn’t occurring.  Any suggestions?  Is
>> it safe to identify one of my four servers as containing up-to-date data,
>> then sever and reinstate replication relationships with the other three?
>>
>> Mike
>>
>>
>>
>>
>>
>>
>> --
>> Manage your subscription for the Freeipa-users mailing list:
>> https://www.redhat.com/mailman/listinfo/freeipa-users
>> Go to http://freeipa.org for more info on the project
>>
>
>
>
>
> --
> Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
> Commercial register: Amtsgericht Muenchen, HRB 153243,
> Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, 
> Eric Shander
>
>
> --
> Manage your subscription for the Freeipa-users mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go to http://freeipa.org for more info on the project
>
-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/

Re: [Freeipa-users] Server replication stopped working

2016-09-26 Thread Ludwig Krispenz


On 09/25/2016 09:35 PM, Youenn PIOLET wrote:

Hi there,

Same issue for me in a my 15 ipa-servers multi-master grid just after 
the update.

The replication is completely broken except on 3/15 nodes.

This is the second time I have to fully reinitialize the whole cluster 
for similar reason. I don't know what to do to clean this mess...
For more information: this cluster has been initialized on a fedora 
4.1.4 more than one year ago then complemetely migrated to Centos 7, 
IPA 4.2.

what is the exact version of 389-ds-base you are running ?

did these errors come out of the blue or are they related to some 
activities ? The csn which is not found has a timestamp of "Thu, 22 Sep 
2016 15:59:08 GMT" did anything happen around this time ?


Example on fr-master03 error logs:

[25/Sep/2016:19:27:31 +] NSMMReplicationPlugin - changelog program 
- agmt="cn=meTofr-master01.domain" (fr-master01:389): CSN 
57e3ffcc0003001a not found, we aren't as up to date, or we purged
[25/Sep/2016:19:27:31 +] NSMMReplicationPlugin - 
agmt="cn=meTofr-master01.domain" (fr-master01:389): Data required to 
update replica has been purged. The replica must be reinitialized.
[25/Sep/2016:19:27:31 +] NSMMReplicationPlugin - 
agmt="cn=meTofr-master01.domain" (fr-master01:389): Incremental update 
failed and requires administrator action

ipa: INFO: The ipactl command was successful
[25/Sep/2016:19:27:35 +] agmt="cn=meTofr-master02.domain" 
(fr-master02:389) - Can't locate CSN 57e3ffcc0003001a in the 
changelog (DB rc=-30988). If replication stops, the consumer may need 
to be reinitialized.
[25/Sep/2016:19:27:35 +] NSMMReplicationPlugin - changelog program 
- agmt="cn=meTofr-master02.domain" (fr-master02:389): CSN 
57e3ffcc0003001a not found, we aren't as up to date, or we purged
[25/Sep/2016:19:27:35 +] NSMMReplicationPlugin - 
agmt="cn=meTofr-master02.domain" (fr-master02:389): Data required to 
update replica has been purged. The replica must be reinitialized.
[25/Sep/2016:19:27:35 +] NSMMReplicationPlugin - 
agmt="cn=meTofr-master02.domain" (fr-master02:389): Incremental update 
failed and requires administrator action


Regards,

--
Youenn Piolet
piole...@gmail.com 
/
/

2016-09-23 17:51 GMT+02:00 Mike Driscoll >:


Hello.  I have four IPA servers replicating in full mesh.  All
four servers are running ipa-server-4.2.0-15.0.1.el7_2.19.x86_64.

This was working for some time but now I see that no replication
is occurring automatically at present.

When I update a user attribute on an IPA server, I see errors like
these:
[22/Sep/2016:16:53:49 -0700] attrlist_replace - attr_replace
(nsslapd-referral, ldap://ldap03.xx.com:389/o%3Dipaca) failed.
[22/Sep/2016:16:58:56 -0700] NSMMReplicationPlugin -
agmt="cn=masterAgreement1-ldap03.xx.com
-pki-tomcat" (ldap03:389):
Incremental update failed and requires administrator action

I can reinitialize without errors.
ipa-csreplica-manage re-initialize --from=ldap01.xx.com

ipa-replica-manage re-initialize --from=ldap01.xx.com

Afterwards I see my attribute (and other) changes are replicated
on each server I re-initialize from.  But subsequently,
replication doesn't seem to be happening.

I reinitialized according to the steps in Table 8.7, "Replication
Errors", but subsequent replication isn't occurring.  Any
suggestions?  Is it safe to identify one of my four servers as
containing up-to-date data, then sever and reinstate replication
relationships with the other three?

Mike






--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users

Go to http://freeipa.org for more info on the project






--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Server replication stopped working

2016-09-25 Thread Youenn PIOLET
Hi there,

Same issue for me in a my 15 ipa-servers multi-master grid just after the
update.
The replication is completely broken except on 3/15 nodes.

This is the second time I have to fully reinitialize the whole cluster for
similar reason. I don't know what to do to clean this mess...
For more information: this cluster has been initialized on a fedora 4.1.4
more than one year ago then complemetely migrated to Centos 7, IPA 4.2.

Example on fr-master03 error logs:

[25/Sep/2016:19:27:31 +] NSMMReplicationPlugin - changelog program -
agmt="cn=meTofr-master01.domain" (fr-master01:389): CSN
57e3ffcc0003001a not found, we aren't as up to date, or we purged
[25/Sep/2016:19:27:31 +] NSMMReplicationPlugin -
agmt="cn=meTofr-master01.domain" (fr-master01:389): Data required to update
replica has been purged. The replica must be reinitialized.
[25/Sep/2016:19:27:31 +] NSMMReplicationPlugin -
agmt="cn=meTofr-master01.domain" (fr-master01:389): Incremental update
failed and requires administrator action
ipa: INFO: The ipactl command was successful
[25/Sep/2016:19:27:35 +] agmt="cn=meTofr-master02.domain"
(fr-master02:389) - Can't locate CSN 57e3ffcc0003001a in the changelog
(DB rc=-30988). If replication stops, the consumer may need to be
reinitialized.
[25/Sep/2016:19:27:35 +] NSMMReplicationPlugin - changelog program -
agmt="cn=meTofr-master02.domain" (fr-master02:389): CSN
57e3ffcc0003001a not found, we aren't as up to date, or we purged
[25/Sep/2016:19:27:35 +] NSMMReplicationPlugin -
agmt="cn=meTofr-master02.domain" (fr-master02:389): Data required to update
replica has been purged. The replica must be reinitialized.
[25/Sep/2016:19:27:35 +] NSMMReplicationPlugin -
agmt="cn=meTofr-master02.domain" (fr-master02:389): Incremental update
failed and requires administrator action

Regards,

--
Youenn Piolet
piole...@gmail.com


2016-09-23 17:51 GMT+02:00 Mike Driscoll :

> Hello.  I have four IPA servers replicating in full mesh.  All four
> servers are running ipa-server-4.2.0-15.0.1.el7_2.19.x86_64.
>
> This was working for some time but now I see that no replication is
> occurring automatically at present.
>
> When I update a user attribute on an IPA server, I see errors like these:
> [22/Sep/2016:16:53:49 -0700] attrlist_replace - attr_replace
> (nsslapd-referral, ldap://ldap03.xx.com:389/o%3Dipaca) failed.
> [22/Sep/2016:16:58:56 -0700] NSMMReplicationPlugin - agmt="cn=
> masterAgreement1-ldap03.xx.com 
> -pki-tomcat" (ldap03:389): Incremental update failed and requires
> administrator action
>
> I can reinitialize without errors.
> ipa-csreplica-manage re-initialize --from=ldap01.xx.com
> 
> ipa-replica-manage re-initialize --from=ldap01.xx.com
> Afterwards I see my attribute (and other) changes are replicated on each
> server I re-initialize from.  But subsequently, replication doesn’t seem to
> be happening.
>
> I reinitialized according to the steps in Table 8.7, “Replication Errors”,
> but subsequent replication isn’t occurring.  Any suggestions?  Is it safe
> to identify one of my four servers as containing up-to-date data, then
> sever and reinstate replication relationships with the other three?
>
> Mike
>
>
>
>
>
>
> --
> Manage your subscription for the Freeipa-users mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go to http://freeipa.org for more info on the project
>
-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project