date:20170519

[Freeipa-users] replicating cn=accounts, dc=ipa, dc=example, dc=com tree to a read-only instance of 389ds on our mailserver

2017-05-19 Thread Bernhard Kneip


Hi guys,

our current setup consists of 3 replicated free-ipa servers in a 
master-master configuration.
What we are currently trying to do, is to add a standalone 389-ds on our 
mailserver which should only readonly-replicate 
cn=accounts,dc=ipa,dc=example,dc=com to enable our mailserver to have a 
local ldap cache (for alias/mailbox mapping in postfix/dovecot) and to 
be able to add a local ldap-addressbook to our mailserver without the 
need to have it on our ipa-servers.


Our environment is:

3 free-ipa servers
(centos7, 389-ds-base.x86_64 1.3.5.10-20.el7_3)

1 Mailserver
(debian stretch, 389-ds 1.3.5.15-2)

What we did do:

Basically following this guide:
https://access.redhat.com/documentation/en-us/red_hat_directory_server/10/html/administration_guide/Managing_Replication-Configuring-Replication-cmd#Configuring-Replication-Suppliers-cmd

on consumer (our mailserver):
...first we created the missing root 
(cn=accounts,dc=ipa,dc=example,dc=com) by hand


# readonly replication manager
dn: cn=readonly replication manager,cn=config
objectClass: inetorgperson
objectClass: person
objectClass: top
cn: readonly replication manager
sn: RORM
userPassword: NotTheRealPassword
passwordExpirationTime: 20380119031407Z
nsIdleTimeout: 0


Replication Entry:

# no dc=ipa in the dn!
dn: cn=replica,cn=dc\=example\,dc\=com,cn=mapping tree,cn=config
changetype: add
objectclass: top
objectclass: nsds5replica
objectclass: extensibleObject
cn: replica
nsds5replicaid: 65535
nsds5replicaroot: cn=accounts,dc=ipa,dc=example,dc=com
nsds5replicatype: 2
nsds5ReplicaPurgeDelay: 604800
nsds5ReplicaBindDN: cn=replication manager,cn=config
nsds5flags: 1


# on supplier (one of our IPA-servers)
# on our IPA-servers, dc=ipa is included
dn: 
cn=accountsToMail,cn=replica,cn=dc\=ipa\,dc\=example\,dc\=com,cn=mapping 
tree,cn=config

objectclass: top
objectclass: nsds5ReplicationAgreement
cn: accounts2hermes
nsds5replicahost: mail.example.com
nsds5replicaport: 389
nsds5ReplicaBindDN: cn=readonly replication manager,cn=config
nsds5replicabindmethod: SIMPLE
nsds5replicaroot: cn=accounts,dc=ipa,dc=example,dc=com
description: replicate cn=accounts from ipa to hermes
nsds5replicatedattributelist: (objectclass=*) $ EXCLUDE 
authorityRevocationList accountUnlockTime memberof
nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE 
accountUnlockTime

nsds5replicacredentials: notTheRealButSameAsAbove
nsds5ReplicaIgnoreMissingChange: once
nsds5BeginReplicaRefresh: start


After some log-entries regarding the schema versions, we stopped the 
consumer and copied the schema from the supplier to the consumer by hand...
This fixed most of the noise in the log, but we are still getting the 
following error:


[18/May/2017:10:23:41.311816674 +0200] NSMMReplicationPlugin - 
agmt="cn=accountsToMail" (mail:389): The remote replica has a different 
database generation ID tha
n the local database.  You may have to reinitialize the remote replica, 
or the local replica.


Of course, we tried to re-initialize the remote-replica by,

dn: 
cn=accountsToMail,cn=replica,cn=dc\=ipa\,dc\=example\,dc\=com,cn=mapping 
tree,cn=config

changetype: modify
replace: nsds5BeginReplicaRefresh
nsds5BeginReplicaRefresh: start


What are we missing?

Best regards,

Bernhard

--
Bernhard Kneip
Systemadministration

E-Mail: bernhard.kn...@isa.de.com
Tel: +49(0)3677/46929-144
Internet: www.isa.de.com

ISA Institut für Serviceautomation GmbH & Co. KG
Ziolkowskistraße 8, 98693 Ilmenau
Amtsgericht Jena, HRA 301735
persönlich haftende Gesellschafterin: ISA GmbH
Amtsgericht Jena, HRB 306708
Geschäftsführer: Dr.-Ing. Walther Spies, Dipl.-Ing. (FH) Peter Mayer

Member of SIELAFF GROUP

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Replica cannot be reinitialized after upgrade

2017-05-19 Thread Ludwig Krispenz



On 05/18/2017 10:13 PM, Goran Marik wrote:

Thanks Ludwig for the suggestion and thanks to Maciej for the confirmation from 
his end. This issue is happening for us for several weeks, so I don’t think 
this is a transient problem.

What is the best way to sanitize the logs without removing useful info before sending them 
your way? Will the files mentioned on 
"https://www.freeipa.org/page/Files_to_be_attached_to_bug_report -> Directory server 
failed" be sufficient?
yes, but we need soem additional info on the replication config and 
state, you could add /etc/dirsrv/slapd-*/dse.ldif

and the result of these query

ldapsearch -o ldif-wrap=no  -D "cn=directory 
manager" ... -b "cn=config" "objectclass=nsds5replica" \* nsds50ruv


But looking again at the csn reorted missing it is from June, 2016. So I 
wonder if this is for an stale/removed replica and cleaning the ruvs 
would help


I’ve also run the ipa_consistency_check script, and the output shows that 
something is indeed wrong with the sync:
“””
FreeIPA servers:inf01inf01inf02inf02STATE
=
Active Users15   15   15   15   OK
Stage Users 0000OK
Preserved Users 3333OK
User Groups 9999OK
Hosts   45   45   45   46   FAIL
Host Groups 7777OK
HBAC Rules  6666OK
SUDO Rules  7777OK
DNS Zones   33   33   33   33   OK
LDAP Conflicts  NO   NO   NO   NO   OK
Ghost Replicas  2222FAIL
Anonymous BIND  YES  YES  YES  YES  OK
Replication Status  inf01.prod 0inf01.dev 0inf01.dev 0inf01.dev 0
 inf02.dev 0inf02.dev 0inf01.prod 0inf01.prod 0
 inf02.prod 0inf02.prod 0inf02.prod 0inf02.dev 0
=
“””

Thanks,
Goran


On May 15, 2017, at 6:35 AM, Ludwig Krispenz  wrote:

The messages you see could be transient messages, and if replication is working 
than this seems to be the case. If not we would need more data to investigate: 
deployment info, relicaIDs of all servers, ruvs, logs,.

Here is some background info: there are some scenarios where a csn could not be 
found in the changelog, eg if updates were aplied on the supplier during a 
total init, they could be part of the data and database ruv, but not in the 
changelog of the initialized replica.
ds did try to use an alternative csn in cases where it could not be found, but 
this had the risk of missing updates, so we decided to change it and make this 
misssing csn a non fatal error, backoff and retry, if another supplier would 
have updated the replica in between, the starting csn could have changed and be 
found. so if the reported missing csns change and replication continues 
everything is ok, although I think the messages should stop at some point.

There is a configuration parameter for a replciation agreement to trigger the 
previous behaviour of picking an alternative csn:
nsds5ReplicaIgnoreMissingChange
with potential values "once", "always".

where "once" just tries to kickstart replication by using another csn and 
"always" changes the default behaviour


On 05/11/2017 06:53 PM, Goran Marik wrote:

Hi,

After an upgrade to Centos 7.3.1611 with “yum update", we started seeing the 
following messages in the logs:
“””
May  9 21:58:28 inf01 ns-slapd[4323]: [09/May/2017:21:58:28.519724479 +] 
NSMMReplicationPlugin - changelog program - 
agmt="cn=cloneAgreement1-inf02.dev.ecobee.com-pki-tomcat" (inf02:389): CSN 
576b34e8000a050f not found, we aren't as up to date, or we purged
May  9 21:58:28 inf01 ns-slapd[4323]: [09/May/2017:21:58:28.550459233 +] 
NSMMReplicationPlugin - 
agmt="cn=cloneAgreement1-inf02.dev.ecobee.com-pki-tomcat" (inf02:389): Data 
required to update replica has been purged from the changelog. The replica must be 
reinitialized.
May  9 21:58:32 inf01 ns-slapd[4323]: [09/May/2017:21:58:32.588245476 +] 
agmt="cn=cloneAgreement1-inf02.dev.ecobee.com-pki-tomcat" (inf02:389) - Can't 
locate CSN 576b34e8000a050f in the changelog (DB rc=-30988). If replication stops, 
the consumer may need to be reinitialized.
May  9 21:58:32 inf01 ns-slapd[4323]: [09/May/2017:21:58:32.611400689 +] 
NSMMReplicationPlugin - changelog program - 
agmt="cn=cloneAgreement1-inf02.dev.ecobee.com-pki-tomcat" (inf02:389): CSN 
576b34e8000a050f not found, we aren't as up to date, or we purged
May  9 21:58:32 inf01 ns-slapd[4323]: [09/May/2017:21:58:32.642226385 +] 
NSMMReplicationPlugin - 
agmt="cn=cloneAgreement1-inf02.dev.ecobee.com-pki-tomcat" (inf02:389): Data 
required to update replica has been

[Freeipa-users] replicating cn=accounts, dc=ipa, dc=example, dc=com tree to a read-only instance of 389ds on our mailserver

2017-05-19 Thread Bernhard Kneip


Hi guys,

our current setup consists of 3 replicated free-ipa servers in a 
master-master configuration.
What we are currently trying to do, is to add a standalone 389-ds on our 
mailserver which should only readonly-replicate 
cn=accounts,dc=ipa,dc=example,dc=com to enable our mailserver to have a 
local ldap cache (for alias/mailbox mapping in postfix/dovecot) and to 
be able to add a local ldap-addressbook to our mailserver without the 
need to have it on our ipa-servers.


Our environment is:

3 free-ipa servers
(centos7, 389-ds-base.x86_64 1.3.5.10-20.el7_3)

1 Mailserver
(debian stretch, 389-ds 1.3.5.15-2)

What we did do:

Basically following this guide:
https://access.redhat.com/documentation/en-us/red_hat_directory_server/10/html/administration_guide/Managing_Replication-Configuring-Replication-cmd#Configuring-Replication-Suppliers-cmd

on consumer (our mailserver):
...first we created the missing root 
(cn=accounts,dc=ipa,dc=example,dc=com) by hand


# readonly replication manager
dn: cn=readonly replication manager,cn=config
objectClass: inetorgperson
objectClass: person
objectClass: top
cn: readonly replication manager
sn: RORM
userPassword: NotTheRealPassword
passwordExpirationTime: 20380119031407Z
nsIdleTimeout: 0


Replication Entry:

# no dc=ipa in the dn!
dn: cn=replica,cn=dc\=example\,dc\=com,cn=mapping tree,cn=config
changetype: add
objectclass: top
objectclass: nsds5replica
objectclass: extensibleObject
cn: replica
nsds5replicaid: 65535
nsds5replicaroot: cn=accounts,dc=ipa,dc=example,dc=com
nsds5replicatype: 2
nsds5ReplicaPurgeDelay: 604800
nsds5ReplicaBindDN: cn=replication manager,cn=config
nsds5flags: 1


# on supplier (one of our IPA-servers)
# on our IPA-servers, dc=ipa is included
dn: 
cn=accountsToMail,cn=replica,cn=dc\=ipa\,dc\=example\,dc\=com,cn=mapping 
tree,cn=config

objectclass: top
objectclass: nsds5ReplicationAgreement
cn: accounts2hermes
nsds5replicahost: mail.example.com
nsds5replicaport: 389
nsds5ReplicaBindDN: cn=readonly replication manager,cn=config
nsds5replicabindmethod: SIMPLE
nsds5replicaroot: cn=accounts,dc=ipa,dc=example,dc=com
description: replicate cn=accounts from ipa to hermes
nsds5replicatedattributelist: (objectclass=*) $ EXCLUDE 
authorityRevocationList accountUnlockTime memberof
nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE 
accountUnlockTime

nsds5replicacredentials: notTheRealButSameAsAbove
nsds5ReplicaIgnoreMissingChange: once
nsds5BeginReplicaRefresh: start


After some log-entries regarding the schema versions, we stopped the 
consumer and copied the schema from the supplier to the consumer by hand...
This fixed most of the noise in the log, but we are still getting the 
following error:


[18/May/2017:10:23:41.311816674 +0200] NSMMReplicationPlugin - 
agmt="cn=accountsToMail" (mail:389): The remote replica has a different 
database generation ID tha
n the local database.  You may have to reinitialize the remote replica, 
or the local replica.


Of course, we tried to re-initialize the remote-replica by,

dn: 
cn=accountsToMail,cn=replica,cn=dc\=ipa\,dc\=example\,dc\=com,cn=mapping 
tree,cn=config

changetype: modify
replace: nsds5BeginReplicaRefresh
nsds5BeginReplicaRefresh: start


What are we missing?

Best regards,

Bernhard

--
Bernhard Kneip
Systemadministration

E-Mail: bernhard.kn...@isa.de.com
Tel: +49(0)3677/46929-144
Internet: www.isa.de.com

ISA Institut für Serviceautomation GmbH & Co. KG
Ziolkowskistraße 8, 98693 Ilmenau
Amtsgericht Jena, HRA 301735
persönlich haftende Gesellschafterin: ISA GmbH
Amtsgericht Jena, HRB 306708
Geschäftsführer: Dr.-Ing. Walther Spies, Dipl.-Ing. (FH) Peter Mayer

Member of SIELAFF GROUP

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Cant locate CSN after yum update

2017-05-19 Thread Christophe TREFOIS

Dear Ludwig,

Thank you for the explanations. Now I understand.

Strangely then, the problem csn was on the replica that we had to reinitialize. 
How could such a csn disappear?

Thanks again for the help. Much appreciated.
Sent from my iPhone

> On 19 May 2017, at 08:47, Ludwig Krispenz  wrote:
> 
> 
>> On 05/18/2017 05:35 PM, Christophe TREFOIS wrote:
>> Dear Ludwig,
>> 
>> Thanks for your help in IRC to guide me in running the right commands.
>> 
>> Here is the output, toto1 and toto2 are CA master, and toto3 and toto4 are 
>> non CA master. The problematic replica was toto3, and after re-init, we 
>> haven’t seen any errors in the log anymore.
>> 
>> https://paste.fedoraproject.org/paste/j8c30CZPyh8rPymjbKSvZF5M1UNdIGYhyRLivL9gydE=
>> 
>> I also ran ipa-replica-manage on all replicas to all replicas, so total of 
>> 16 command, and found all of them reported “incremental update succeeded”.
>> 
>> As discussed, I’m not sure what I’m looking at with the RUV stuff above, and 
>> any explanation for a newcomer to ldap / ds / freeipa would be greatly 
>> appreciated.
> ok, here is a quick explanation of the csn/ruv stuff. 
> 
> each change applied on a server gets a CSN (change sequence number), it 
> basically consists of a timestamp and an identifier of the replica where it 
> was originally applied, so in 59095fe1000b0012 there is a time stamp: 
> 59095fe1 and a replicaid: 0012 == 18, the rest of the csn isused to serialize 
> csns within the one second resolution of a timestamp.
> a change is applied to the main database and written to the changelog, with 
> the csn as key.
> 
> now each replica keeps track of the latest csn it has seen for each 
> replicaID, so you get a vector of max csns, this is called RUV (replica 
> update vector).
> In a replication session, the supplier compares its own ruv with the ruv of 
> the consumer and so decides if it has changes which the consumer has not yet 
> seen.
> based on the consumer ruv it determines the start csn to send updates.
> 
> 
>> 
>> Thanks a lot for your help!
>> 
>> Kind regards,
>> Christophe aka Trefex
>> 
>>> On 18 May 2017, at 17:04, Christophe TREFOIS  
>>> wrote:
>>> 
>>> Hi Ludwig,
>>> 
>>> Since we were scared, we did a full re-init of that specific replica from 
>>> the CA master, and it looks like the issue is not appearing anymore.
>>> 
>>> Is this sufficient, or should we still investigate ?
>>> 
>>> Thanks for your help!
>>> Christophe
>>> -- 
>>> 
>>> Dr Christophe Trefois, Dipl.-Ing.  
>>> Technical Specialist / Post-Doc
>>> 
>>> UNIVERSITÉ DU LUXEMBOURG
>>> 
>>> LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE
>>> Campus Belval | House of Biomedicine  
>>> 6, avenue du Swing 
>>> L-4367 Belvaux  
>>> T: +352 46 66 44 6124 
>>> F: +352 46 66 44 6949  
>>> http://www.uni.lu/lcsb
>>> 
>>> 
>>> 
>>> 
>>> 
>>> This message is confidential and may contain privileged information. 
>>> It is intended for the named recipient only. 
>>> If you receive it in error please notify me and permanently delete the 
>>> original message and any copies. 
>>> 
>>>   
>>> 
 On 18 May 2017, at 16:11, Ludwig Krispenz  wrote:
 
 hi,
 
 there was a change that in the case of a missing csn ds would not silently 
 use a "close" one and continue, but log an error, backoff and retry - 
 after updates on other masters the staring csn coudl change and 
 replication continue.
 
 Now, in your case the csn reported missing: 59095fe1000b0012
 has a time stamp from May,3rd, so it could very well be correct that this 
 csn is no longer found in the changelog.
 
 To continue analysis, could you provide the replicaids of all your current 
 replicas, and which is the replicaid of the sever logging the change and 
 the ruvs of the replicas from all servers.
 ldapsearch   -D "cn=directory manager"  -b cn=config 
 "objectclass=nsds5replica" nsds50ruv
 
 Regards,
 Ludwig
 
> On 05/18/2017 03:09 PM, Christophe TREFOIS wrote:
> Hi all,
> 
> Did a yum update on one of my replicas, non CA master, and upgrade was 
> successful (ipupgrade.log) said so. 
> 
> 
> Hwoever, now every few seconds I get the following message. 
> https://paste.fedoraproject.org/paste/wS4x9KvD3EB0gv2HAsj6X15M1UNdIGYhyRLivL9gydE=
> 
> Does anybody know how to proceed and if this is important?
> ipa-replica-manage says, backing off, retrying later, so not sure if 
> replication happens successfully or not and what to do ??
> 
> Setup: CentOS 7.3 all up-to-date, 2 CA master, 2 non CA master in diamond 
> replication.
> 
> Remaining replicas were upgraded today as well, and don’t seem to 
> complain. Only 1 of them complains.
> 
> 389-ds-base-libs-1.3.5.10-20.el7_3.x86_64
> 389-ds-base-1.3.5.10-20.el7_3.x86_64
> 
> 
> [root@lums3 ~]#

Re: [Freeipa-users] Cant locate CSN after yum update

2017-05-19 Thread Ludwig Krispenz



On 05/18/2017 05:35 PM, Christophe TREFOIS wrote:

Dear Ludwig,

Thanks for your help in IRC to guide me in running the right commands.

Here is the output, toto1 and toto2 are CA master, and toto3 and toto4 
are non CA master. The problematic replica was toto3, and after 
re-init, we haven’t seen any errors in the log anymore.


https://paste.fedoraproject.org/paste/j8c30CZPyh8rPymjbKSvZF5M1UNdIGYhyRLivL9gydE=

I also ran ipa-replica-manage on all replicas to all replicas, so 
total of 16 command, and found all of them reported “incremental 
update succeeded”.


As discussed, I’m not sure what I’m looking at with the RUV stuff 
above, and any explanation for a newcomer to ldap / ds / freeipa would 
be greatly appreciated.

ok, here is a quick explanation of the csn/ruv stuff.

each change applied on a server gets a CSN (change sequence number), it 
basically consists of a timestamp and an identifier of the replica where 
it was originally applied, so in 59095fe1000b0012 there is a time 
stamp: 59095fe1 and a replicaid: 0012 == 18, the rest of the csn isused 
to serialize csns within the one second resolution of a timestamp.
a change is applied to the main database and written to the changelog, 
with the csn as key.


now each replica keeps track of the latest csn it has seen for each 
replicaID, so you get a vector of max csns, this is called RUV (replica 
update vector).
In a replication session, the supplier compares its own ruv with the ruv 
of the consumer and so decides if it has changes which the consumer has 
not yet seen.

based on the consumer ruv it determines the start csn to send updates.




Thanks a lot for your help!

Kind regards,
Christophe aka Trefex

On 18 May 2017, at 17:04, Christophe TREFOIS 
> wrote:


Hi Ludwig,

Since we were scared, we did a full re-init of that specific replica 
from the CA master, and it looks like the issue is not appearing anymore.


Is this sufficient, or should we still investigate ?

Thanks for your help!
Christophe

--

Dr Christophe Trefois, Dipl.-Ing.
Technical Specialist / Post-Doc

UNIVERSITÉ DU LUXEMBOURG

LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE
Campus Belval | House of Biomedicine
6, avenue du Swing
L-4367 Belvaux
T:+352 46 66 44 6124
F:+352 46 66 44 6949
http://www.uni.lu/lcsb

Facebook Twitter 
Google Plus 
Linkedin 
skype




This message is confidential and may contain privileged information.
It is intended for the named recipient only.
If you receive it in error please notify me and permanently delete 
the original message and any copies.




On 18 May 2017, at 16:11, Ludwig Krispenz > wrote:


hi,

there was a change that in the case of a missing csn ds would not 
silently use a "close" one and continue, but log an error, backoff 
and retry - after updates on other masters the staring csn coudl 
change and replication continue.


Now, in your case the csn reported missing: 59095fe1000b0012
has a time stamp from May,3rd, so it could very well be correct that 
this csn is no longer found in the changelog.


To continue analysis, could you provide the replicaids of all your 
current replicas, and which is the replicaid of the sever logging 
the change and the ruvs of the replicas from all servers.
ldapsearch   -D "cn=directory manager"  -b cn=config 
"objectclass=nsds5replica" nsds50ruv


Regards,
Ludwig

On 05/18/2017 03:09 PM, Christophe TREFOIS wrote:

Hi all,

Did a yum update on one of my replicas, non CA master, and upgrade 
was successful (ipupgrade.log) said so.



Hwoever, now every few seconds I get the following message. 
https://paste.fedoraproject.org/paste/wS4x9KvD3EB0gv2HAsj6X15M1UNdIGYhyRLivL9gydE=


Does anybody know how to proceed and if this is important?
ipa-replica-manage says, backing off, retrying later, so not sure 
if replication happens successfully or not and what to do ??


Setup: CentOS 7.3 all up-to-date, 2 CA master, 2 non CA master in 
diamond replication.


Remaining replicas were upgraded today as well, and don’t seem to 
complain. Only 1 of them complains.


389-ds-base-libs-1.3.5.10-20.el7_3.x86_64
389-ds-base-1.3.5.10-20.el7_3.x86_64


[root@lums3 ~]# rpm -qa | grep ipa
libipa_hbac-1.14.0-43.el7_3.14.x86_64
python-iniparse-0.4-9.el7.noarch
ipa-admintools-4.4.0-14.el7.centos.7.noarch
python2-ipaserver-4.4.0-14.el7.centos.7.noarch
python2-ipalib-4.4.0-14.el7.centos.7.noarch
sssd-ipa-1.14.0-43.el7_3.14.x86_64
python-ipaddress-1.0.16-2.el7.noarch
python2-ipaclient-4.4.0-14.el7.centos.7.noarch
ipa-server-common-4.4.0-14.el7.centos.7.noarch
ipa-client-common-4.4.0-14.el7.centos.7.noarch
ipa-client-4.4.0-14.el7.centos.7.x86_64
ipa-common-4.4.0-14.el7.centos.7.noarch
python-libipa_hbac-1.14.0-43.el7_3.14.x86_64

[Freeipa-users] replicating cn=accounts, dc=ipa, dc=example, dc=com tree to a read-only instance of 389ds on our mailserver

Re: [Freeipa-users] Replica cannot be reinitialized after upgrade

[Freeipa-users] replicating cn=accounts, dc=ipa, dc=example, dc=com tree to a read-only instance of 389ds on our mailserver

Re: [Freeipa-users] Cant locate CSN after yum update

Re: [Freeipa-users] Cant locate CSN after yum update

5 matches

Site Navigation

Mail list logo

Footer information