Re: [Freeipa-users] CSN not found

2016-11-08 Thread lejeczek



On 03/11/16 19:58, Mark Reynolds wrote:

dbscan -f /var/lib/dirsrv/slapd-INSTANCE/db/changelogdb

>results of above scan do not look like that CSN form reported in
>dirsrv's error log, it is:
>..
>=116156
>=116157
>=116158
>..

That doesn't look quite right,  Just to confirm you should be doing
something like

dbscan -f
/var/lib/dirsrv/slapd-master_1/db/changelogdb/fe665489-a13011e6-acbab8c1-43b12a38_581a3c410001.db
| grep 581b120f00050004
I don't see any xx.db in 
/var/lib/dirsrv/slapd-master_1/db/changelogdb

but there are these:

16c9da9e-a54611e6-80ab82b9-81e5c5a8_574596220060.db
16c9da9e-a54611e6-80ab82b9-81e5c5a8.sema
DBVERSION
e71ad28c-a54511e6-80ab82b9-81e5c5a8_574595c80004.db
e71ad28c-a54511e6-80ab82b9-81e5c5a8.sema

in /var/lib/dirsrv/slapd-master_1/cldb and if I scant those:

cldb]$ for _F in .db; do dbscan -f $_F | grep 
57480d6d0025; done


there is nothing (on the replica that complains but also 
nothing on all members)


cldb]$ ll ../db/changelog/
total 2260
-rw---. 1 dirsrv dirsrv   16384 Nov  8 00:02 aci.db
-rw---. 1 dirsrv dirsrv   40960 Nov  8 15:52 ancestorid.db
-rw---. 1 dirsrv dirsrv   40960 Nov  8 15:52 changenumber.db
-rw---. 1 dirsrv dirsrv   16384 Nov  8 00:02 cn.db
-rw---. 1 dirsrv dirsrv  51 Nov  8 00:02 DBVERSION
-rw---. 1 dirsrv dirsrv  303104 Nov  8 15:52 entryrdn.db
-rw---. 1 dirsrv dirsrv   40960 Nov  8 15:52 entryusn.db
-rw---. 1 dirsrv dirsrv 1523712 Nov  8 15:52 id2entry.db
-rw---. 1 dirsrv dirsrv   90112 Nov  8 15:52 nsuniqueid.db
-rw---. 1 dirsrv dirsrv   16384 Nov  8 15:52 
numsubordinates.db

-rw---. 1 dirsrv dirsrv   90112 Nov  8 15:52 objectclass.db
-rw---. 1 dirsrv dirsrv   40960 Nov  8 15:52 parentid.db
-rw---. 1 dirsrv dirsrv   16384 Nov  8 00:02 seeAlso.db
-rw---. 1 dirsrv dirsrv   65536 Nov  8 15:52 
targetuniqueid.db


it's centOS 7 with IPA 
ipa-server-4.2.0-15.0.1.el7.centos.19.x86_64



>>
>>What about the access logs?  Do you see the CSN there?

Did you check the DS access logs??


--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project


Re: [Freeipa-users] CSN not found

2016-11-03 Thread Mark Reynolds


On 11/03/2016 12:49 PM, lejeczek wrote:
>
>
> On 03/11/16 14:16, Mark Reynolds wrote:
>>
>> On 11/03/2016 09:42 AM, lejeczek wrote:
>>> hi everybody
>>>
>>> my three IPAs have gone haywire, two things I recall: one - one server
>>> was on ScientificL with slightly lower minor version of IPA, two -
>>> another server (of the two identical CEntOSes) had skewed time.
>>> Not all there servers are in time-sync and all run same version of IPA
> here I meant: Now all there
>>> but replication broke with errors like:
>>>
>>>
>>> $ ipa-replica-manage re-initialize --from rider --force
>>>
>>> ..
>>> [03/Nov/2016:13:21:08 +] NSACLPlugin - The ACL target
>>> cn=casigningcert
>>> cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=dc=xx,dc=xx,dc=dc=xx,dc=xx,dc=x
>>>
>>> does not exist
>>> [03/Nov/2016:13:21:08 +] NSACLPlugin - The ACL target
>>> cn=casigningcert
>>> cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=dc=xx,dc=xx,dc=dc=xx,dc=xx,dc=x
>>>
>>> does not exist
>>> [03/Nov/2016:13:21:09 +] agmt="cn=meToswir.xx.xx.xx.xx.x"
>>> (swir:389) - Can't locate CSN 581b120f00050004 in the changelog
>>> (DB rc=-30988). If replication stops, the consumer may need to be
>>> reinitialized.
>>> [03/Nov/2016:13:21:09 +] NSMMReplicationPlugin - changelog program
>>> - agmt="cn=meToswir.xx.xx.xx.xx.x" (swir:389): CSN
>>> 581b120f00050004 not found, we aren't as up to date, or we purged
>>> [03/Nov/2016:13:21:09 +] NSMMReplicationPlugin -
>>> agmt="cn=meToswir.xx.xx.xx.xx.x" (swir:389): Data required to update
>>> replica has been purged. The replica must be reinitialized.
>>> [03/Nov/2016:13:21:09 +] NSMMReplicationPlugin -
>>> agmt="cn=meToswir.xx.xx.xx.xx.x" (swir:389): Incremental update failed
>>> and requires administrator action
>>>
>>> I did dbscan -f /var.../cb941db on all three servers and greped
>>> but cannot see that 581b120f00050004
>>>
>>> where to troubleshoot?
>> What version of 389 do you have:
>>
>> rpm -qa | grep 389-ds-base
>>
>> Did you check the changelog database for 581b120f00050004:
>>
>> dbscan -f /var/lib/dirsrv/slapd-INSTANCE/db/changelogdb
> results of above scan do not look like that CSN form reported in
> dirsrv's error log, it is:
> ..
> =116156
> =116157
> =116158
> ..
That doesn't look quite right,  Just to confirm you should be doing
something like

dbscan -f
/var/lib/dirsrv/slapd-master_1/db/changelogdb/fe665489-a13011e6-acbab8c1-43b12a38_581a3c410001.db
| grep 581b120f00050004
>>
>> What about the access logs?  Do you see the CSN there?
Did you check the DS access logs??
>>
>> I've seen this issue before where a CSN is missing, which breaks the
>> replication agreements, but the CSN does get added to the changelog
>> after a few seconds.  The only way to fix replication is to restart the
>> server, or disable/enable the replication agreements(basically restart
>> them).
> restarting is not possible for the systemctl start ipa fails, though
> system start dirsrv@... succeeds
I meant restart the directory server, not freeipa:

# restart-dirsrv
> what would be correct process of removing repl agreements? 
You don't delete them, you just disable and re-enable them:

https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/10.1/html/Administration_Guide/disabling-replication.html


> I'm trying disconnect/del but am not sure if this is the way.
>
>> Thanks,
>> Mark
>>> many thanks.
>>> L
>>>
>

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project


Re: [Freeipa-users] CSN not found

2016-11-03 Thread lejeczek



On 03/11/16 14:16, Mark Reynolds wrote:


On 11/03/2016 09:42 AM, lejeczek wrote:

hi everybody

my three IPAs have gone haywire, two things I recall: one - one server
was on ScientificL with slightly lower minor version of IPA, two -
another server (of the two identical CEntOSes) had skewed time.
Not all there servers are in time-sync and all run same version of IPA

here I meant: Now all there

but replication broke with errors like:


$ ipa-replica-manage re-initialize --from rider --force

..
[03/Nov/2016:13:21:08 +] NSACLPlugin - The ACL target
cn=casigningcert
cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=dc=xx,dc=xx,dc=dc=xx,dc=xx,dc=x
does not exist
[03/Nov/2016:13:21:08 +] NSACLPlugin - The ACL target
cn=casigningcert
cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=dc=xx,dc=xx,dc=dc=xx,dc=xx,dc=x
does not exist
[03/Nov/2016:13:21:09 +] agmt="cn=meToswir.xx.xx.xx.xx.x"
(swir:389) - Can't locate CSN 581b120f00050004 in the changelog
(DB rc=-30988). If replication stops, the consumer may need to be
reinitialized.
[03/Nov/2016:13:21:09 +] NSMMReplicationPlugin - changelog program
- agmt="cn=meToswir.xx.xx.xx.xx.x" (swir:389): CSN
581b120f00050004 not found, we aren't as up to date, or we purged
[03/Nov/2016:13:21:09 +] NSMMReplicationPlugin -
agmt="cn=meToswir.xx.xx.xx.xx.x" (swir:389): Data required to update
replica has been purged. The replica must be reinitialized.
[03/Nov/2016:13:21:09 +] NSMMReplicationPlugin -
agmt="cn=meToswir.xx.xx.xx.xx.x" (swir:389): Incremental update failed
and requires administrator action

I did dbscan -f /var.../cb941db on all three servers and greped
but cannot see that 581b120f00050004

where to troubleshoot?

What version of 389 do you have:

rpm -qa | grep 389-ds-base

Did you check the changelog database for 581b120f00050004:

dbscan -f /var/lib/dirsrv/slapd-INSTANCE/db/changelogdb
results of above scan do not look like that CSN form 
reported in dirsrv's error log, it is:

..
=116156
=116157
=116158
..


What about the access logs?  Do you see the CSN there?

I've seen this issue before where a CSN is missing, which breaks the
replication agreements, but the CSN does get added to the changelog
after a few seconds.  The only way to fix replication is to restart the
server, or disable/enable the replication agreements(basically restart
them).
restarting is not possible for the systemctl start ipa 
fails, though system start dirsrv@... succeeds
what would be correct process of removing repl agreements? 
I'm trying disconnect/del but am not sure if this is the way.



Thanks,
Mark

many thanks.
L



--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project


Re: [Freeipa-users] CSN not found

2016-11-03 Thread Mark Reynolds


On 11/03/2016 09:42 AM, lejeczek wrote:
> hi everybody
>
> my three IPAs have gone haywire, two things I recall: one - one server
> was on ScientificL with slightly lower minor version of IPA, two -
> another server (of the two identical CEntOSes) had skewed time.
> Not all there servers are in time-sync and all run same version of IPA
> but replication broke with errors like:
>
>
> $ ipa-replica-manage re-initialize --from rider --force
>
> ..
> [03/Nov/2016:13:21:08 +] NSACLPlugin - The ACL target
> cn=casigningcert
> cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=dc=xx,dc=xx,dc=dc=xx,dc=xx,dc=x
> does not exist
> [03/Nov/2016:13:21:08 +] NSACLPlugin - The ACL target
> cn=casigningcert
> cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=dc=xx,dc=xx,dc=dc=xx,dc=xx,dc=x
> does not exist
> [03/Nov/2016:13:21:09 +] agmt="cn=meToswir.xx.xx.xx.xx.x"
> (swir:389) - Can't locate CSN 581b120f00050004 in the changelog
> (DB rc=-30988). If replication stops, the consumer may need to be
> reinitialized.
> [03/Nov/2016:13:21:09 +] NSMMReplicationPlugin - changelog program
> - agmt="cn=meToswir.xx.xx.xx.xx.x" (swir:389): CSN
> 581b120f00050004 not found, we aren't as up to date, or we purged
> [03/Nov/2016:13:21:09 +] NSMMReplicationPlugin -
> agmt="cn=meToswir.xx.xx.xx.xx.x" (swir:389): Data required to update
> replica has been purged. The replica must be reinitialized.
> [03/Nov/2016:13:21:09 +] NSMMReplicationPlugin -
> agmt="cn=meToswir.xx.xx.xx.xx.x" (swir:389): Incremental update failed
> and requires administrator action
>
> I did dbscan -f /var.../cb941db on all three servers and greped
> but cannot see that 581b120f00050004
>
> where to troubleshoot?
What version of 389 do you have:

rpm -qa | grep 389-ds-base

Did you check the changelog database for 581b120f00050004:

dbscan -f /var/lib/dirsrv/slapd-INSTANCE/db/changelogdb

What about the access logs?  Do you see the CSN there?

I've seen this issue before where a CSN is missing, which breaks the
replication agreements, but the CSN does get added to the changelog
after a few seconds.  The only way to fix replication is to restart the
server, or disable/enable the replication agreements(basically restart
them).

Thanks,
Mark
> many thanks.
> L
>

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project


[Freeipa-users] CSN not found

2016-11-03 Thread lejeczek

hi everybody

my three IPAs have gone haywire, two things I recall: one - 
one server was on ScientificL with slightly lower minor 
version of IPA, two - another server (of the two identical 
CEntOSes) had skewed time.
Not all there servers are in time-sync and all run same 
version of IPA but replication broke with errors like:



$ ipa-replica-manage re-initialize --from rider --force

..
[03/Nov/2016:13:21:08 +] NSACLPlugin - The ACL target 
cn=casigningcert 
cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=dc=xx,dc=xx,dc=dc=xx,dc=xx,dc=x 
does not exist
[03/Nov/2016:13:21:08 +] NSACLPlugin - The ACL target 
cn=casigningcert 
cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=dc=xx,dc=xx,dc=dc=xx,dc=xx,dc=x 
does not exist
[03/Nov/2016:13:21:09 +] 
agmt="cn=meToswir.xx.xx.xx.xx.x" (swir:389) - Can't locate 
CSN 581b120f00050004 in the changelog (DB rc=-30988). If 
replication stops, the consumer may need to be reinitialized.
[03/Nov/2016:13:21:09 +] NSMMReplicationPlugin - 
changelog program - agmt="cn=meToswir.xx.xx.xx.xx.x" 
(swir:389): CSN 581b120f00050004 not found, we aren't as 
up to date, or we purged
[03/Nov/2016:13:21:09 +] NSMMReplicationPlugin - 
agmt="cn=meToswir.xx.xx.xx.xx.x" (swir:389): Data required 
to update replica has been purged. The replica must be 
reinitialized.
[03/Nov/2016:13:21:09 +] NSMMReplicationPlugin - 
agmt="cn=meToswir.xx.xx.xx.xx.x" (swir:389): Incremental 
update failed and requires administrator action


I did dbscan -f /var.../cb941db on all three servers and 
greped but cannot see that 581b120f00050004


where to troubleshoot?
many thanks.
L

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project