Re: [Freeipa-users] Replica issue / Certificate Authority

[Yohan JAROSZ]

Hi

@Fraser,
tried the commands and certificates matched in both cases.


@everyone
I tried to look a little bit in the code, and the only references I saw are in
https://github.com/freeipa/freeipa/blob/master/install/certmonger/dogtag-ipa-ca-renew-agent-submit
 (4 references)
And the only one that could fit is this one:
https://github.com/freeipa/freeipa/blob/master/install/certmonger/dogtag-ipa-ca-renew-agent-submit#L142
as our cookie seems to be empty (ca-error: Invalid cookie: '')
and this is the only condition of the 4 that does only test for «  None », the 
other 3 are testing for None, empty strings, … and it should be false.

meaning that somehow the cookie is set somewhere but with no value?

Anyway, do you think it can impact our setup?
Instead of trying to resolve the issue, we could also delete this replica and 
replicate a new one instead?

What do you think?



Yohan
Doing the following up for Christophe.



On 05 Jan 2017, at 07:33, Fraser Tweedale 
mailto:ftwee...@redhat.com>> wrote:

On Wed, Jan 04, 2017 at 01:19:19PM +, Christophe TREFOIS wrote:
Hi Florence,

I did what you said, and then the status went to CA_WORKING. Then I restart ipa 
and certmonger and the status went to CA_UNREACHABLE.
Then i did “resubmit” again and now the status is back to MONITORING, but the 
cookie error is back.

Any advice?

I have encountered the cookie error before. IIRC it was caused by
authn certs in Dogtag user entries not matching the client certs
used.

Check the following entries:

1. ``ldapsearch -LLL -D cn=directory\ manager -w4me2Test \
  -b uid=pkidbuser,ou=people,o=ipaca userCertificate``

  should match

  ``certutil -d /etc/pki/pki-tomcat/alias -L -n "subsystemCert cert-pki-ca"``

2. ``ldapsearch -LLL -D cn=directory\ manager -w4me2Test \
  -b uid=ipara,ou=people,o=ipaca userCertificate``

  should match

  ``certutil -d /etc/httpd/alias -L -n "ipaCert"``

If either of these do not match, update LDAP with what is in the
certificate databases (a.k.a. NSSDBs).  Ensure all certs are
non-expired, etc.

HTH,
Fraser


[root@lums3 ~]# getcert list -n ipaCert
Number of certificates and requests being tracked: 8.
Request ID '20161216025136':
status: MONITORING
ca-error: Invalid cookie: ''
stuck: no
key pair storage: 
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS 
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: 
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS 
Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=UNI.LU
subject: CN=IPA RA,O=UNI.LU
expires: 2018-12-16 03:13:48 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes

--

Dr Christophe Trefois, Dipl.-Ing.
Technical Specialist / Post-Doc

UNIVERSITÉ DU LUXEMBOURG

LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE
Campus Belval | House of Biomedicine
6, avenue du Swing
L-4367 Belvaux
T: +352 46 66 44 6124
F: +352 46 66 44 6949
http://www.uni.lu/lcsb 
      
   
   
>

This message is confidential and may contain privileged information.
It is intended for the named recipient only.
If you receive it in error please notify me and permanently delete the original 
message and any copies.




On 4 Jan 2017, at 13:49, Florence Blanc-Renaud 
mailto:f...@redhat.com>> wrote:

getcert resubmit -i 

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Replica issue / Certificate Authority

[Fraser Tweedale]

On Wed, Jan 04, 2017 at 01:19:19PM +, Christophe TREFOIS wrote:
> Hi Florence,
> 
> I did what you said, and then the status went to CA_WORKING. Then I restart 
> ipa and certmonger and the status went to CA_UNREACHABLE.
> Then i did “resubmit” again and now the status is back to MONITORING, but the 
> cookie error is back.
> 
> Any advice?
> 
I have encountered the cookie error before. IIRC it was caused by
authn certs in Dogtag user entries not matching the client certs
used.

Check the following entries:

1. ``ldapsearch -LLL -D cn=directory\ manager -w4me2Test \
   -b uid=pkidbuser,ou=people,o=ipaca userCertificate``

   should match

   ``certutil -d /etc/pki/pki-tomcat/alias -L -n "subsystemCert cert-pki-ca"``

2. ``ldapsearch -LLL -D cn=directory\ manager -w4me2Test \
   -b uid=ipara,ou=people,o=ipaca userCertificate``

   should match

   ``certutil -d /etc/httpd/alias -L -n "ipaCert"``

If either of these do not match, update LDAP with what is in the
certificate databases (a.k.a. NSSDBs).  Ensure all certs are
non-expired, etc.

HTH,
Fraser


> [root@lums3 ~]# getcert list -n ipaCert
> Number of certificates and requests being tracked: 8.
> Request ID '20161216025136':
>   status: MONITORING
>   ca-error: Invalid cookie: ''
>   stuck: no
>   key pair storage: 
> type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS 
> Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
>   certificate: 
> type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS 
> Certificate DB'
>   CA: dogtag-ipa-ca-renew-agent
>   issuer: CN=Certificate Authority,O=UNI.LU
>   subject: CN=IPA RA,O=UNI.LU
>   expires: 2018-12-16 03:13:48 UTC
>   key usage: 
> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>   eku: id-kp-serverAuth,id-kp-clientAuth
>   pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
>   post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
>   track: yes
>   auto-renew: yes
> 
> -- 
> 
> Dr Christophe Trefois, Dipl.-Ing.  
> Technical Specialist / Post-Doc
> 
> UNIVERSITÉ DU LUXEMBOURG
> 
> LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE
> Campus Belval | House of Biomedicine  
> 6, avenue du Swing 
> L-4367 Belvaux  
> T: +352 46 66 44 6124 
> F: +352 46 66 44 6949  
> http://www.uni.lu/lcsb 
>        
>    
>    
> 
> This message is confidential and may contain privileged information. 
> It is intended for the named recipient only. 
> If you receive it in error please notify me and permanently delete the 
> original message and any copies. 
> 
> 
>   
> 
> > On 4 Jan 2017, at 13:49, Florence Blanc-Renaud  wrote:
> > 
> > getcert resubmit -i 
> 


-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Replica issue / Certificate Authority

[Christophe TREFOIS]

To all,

So to recap, if I hit resubmit once, I get a CA_WORKING, if I do it immediately 
after again, I get a MONITORING, but the “ca-error: Invalid cookie” comes back.

How can I get a valid cookie back?

Thanks for your help,
Christophe

> On 4 Jan 2017, at 14:19, Christophe TREFOIS  wrote:
> 
> Hi Florence,
> 
> I did what you said, and then the status went to CA_WORKING. Then I restart 
> ipa and certmonger and the status went to CA_UNREACHABLE.
> Then i did “resubmit” again and now the status is back to MONITORING, but the 
> cookie error is back.
> 
> Any advice?
> 
> [root@lums3 ~]# getcert list -n ipaCert
> Number of certificates and requests being tracked: 8.
> Request ID '20161216025136':
>   status: MONITORING
>   ca-error: Invalid cookie: ''
>   stuck: no
>   key pair storage: 
> type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS 
> Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
>   certificate: 
> type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS 
> Certificate DB'
>   CA: dogtag-ipa-ca-renew-agent
>   issuer: CN=Certificate Authority,O=UNI.LU
>   subject: CN=IPA RA,O=UNI.LU
>   expires: 2018-12-16 03:13:48 UTC
>   key usage: 
> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>   eku: id-kp-serverAuth,id-kp-clientAuth
>   pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
>   post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
>   track: yes
>   auto-renew: yes
> 
> -- 
> 
> Dr Christophe Trefois, Dipl.-Ing.  
> Technical Specialist / Post-Doc
> 
> UNIVERSITÉ DU LUXEMBOURG
> 
> LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE
> Campus Belval | House of Biomedicine  
> 6, avenue du Swing 
> L-4367 Belvaux  
> T: +352 46 66 44 6124 
> F: +352 46 66 44 6949  
> http://www.uni.lu/lcsb 
>        
>    
>    
> 
> 
> This message is confidential and may contain privileged information. 
> It is intended for the named recipient only. 
> If you receive it in error please notify me and permanently delete the 
> original message and any copies. 
> 
> 
>   
> 
>> On 4 Jan 2017, at 13:49, Florence Blanc-Renaud > > wrote:
>> 
>> getcert resubmit -i 
> 
> -- 
> Manage your subscription for the Freeipa-users mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go to http://freeipa.org for more info on the project



smime.p7s
Description: S/MIME cryptographic signature
-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Replica issue / Certificate Authority

[Christophe TREFOIS]

Hi Florence,

I did what you said, and then the status went to CA_WORKING. Then I restart ipa 
and certmonger and the status went to CA_UNREACHABLE.
Then i did “resubmit” again and now the status is back to MONITORING, but the 
cookie error is back.

Any advice?

[root@lums3 ~]# getcert list -n ipaCert
Number of certificates and requests being tracked: 8.
Request ID '20161216025136':
status: MONITORING
ca-error: Invalid cookie: ''
stuck: no
key pair storage: 
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS 
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: 
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS 
Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=UNI.LU
subject: CN=IPA RA,O=UNI.LU
expires: 2018-12-16 03:13:48 UTC
key usage: 
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes

-- 

Dr Christophe Trefois, Dipl.-Ing.  
Technical Specialist / Post-Doc

UNIVERSITÉ DU LUXEMBOURG

LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE
Campus Belval | House of Biomedicine  
6, avenue du Swing 
L-4367 Belvaux  
T: +352 46 66 44 6124 
F: +352 46 66 44 6949  
http://www.uni.lu/lcsb 
       
   
   

This message is confidential and may contain privileged information. 
It is intended for the named recipient only. 
If you receive it in error please notify me and permanently delete the original 
message and any copies. 


  

> On 4 Jan 2017, at 13:49, Florence Blanc-Renaud  wrote:
> 
> getcert resubmit -i 



smime.p7s
Description: S/MIME cryptographic signature
-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Replica issue / Certificate Authority

[Christophe TREFOIS]

Hi Flo,

The id needed to execute that command would come from where exactly? Is it the 
one from getcert list -n ipaCert?

Thanks
Christophe 

Sent from my iPhone

> On 4 Jan 2017, at 13:49, Florence Blanc-Renaud  wrote:
> 
>> On 01/04/2017 12:41 PM, Christophe TREFOIS wrote:
>> Hi Fraser,
>> 
>> We encountered the same issue. We exported the certificate from a "good"
>> replica, using certutil.
>> We then used certutil -A -n ipaCert -d /etc/httpd/alias/ -i
>> /opt/sysadmin/cacert.crt -a -t CT,C on the bad server and then restarted
>> ipa, and certmonger.
>> 
>> Now, the certificate is correct both using the certutil command and the
>> getcert list -n ipaCert command.
>> 
>> However, we see an "ca-error: Invalid cookie: ''  "  in the output of
>> getcert list -n ipaCert.
>> 
> Hi Christophe,
> 
> is this error displayed on the renewal master? If not, you can run
> $ getcert resubmit -i 
> and the error should go away. On non-renewal master, resubmit downloads the 
> certificate from LDAP (it does not ask for renewal), meaning that this 
> operation cannot be harmful.
> 
> To know which server is the renewal master:
> $ ldapsearch -h localhost -p 389 -D 'cn=Directory Manager' -w password -b 
> "cn=masters,cn=ipa,cn=etc,$BASEDN" 
> '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn
> dn: cn=CA,cn=server.example.com,cn=masters,cn=ipa,cn=etc,$BASEDN
> 
> => the renewal master is server.example.com
> 
> HTH,
> Flo
> 
>> Did we import the certificate correctly and should we worry about this
>> ca-error?
>> It seems replication is going fine, and also ipa-server-upgrade completes
>> successfully when run manually (whereas it failed previously from the same
>> error as in this thread)
>> 
>> Thanks for any pointers on how to clean the issue up properly,
>> Kind regards,
>> 
>> Christophe
>> 
>>> -Original Message-
>>> From: freeipa-users-boun...@redhat.com [mailto:freeipa-users-
>>> boun...@redhat.com] On Behalf Of Fraser Tweedale
>>> Sent: lundi 19 décembre 2016 00:11
>>> To: Christopher Young 
>>> Cc: freeipa-users@redhat.com
>>> Subject: Re: [Freeipa-users] Replica issue / Certificate Authority
>>> 
>>>> On Fri, Dec 16, 2016 at 05:29:18PM -0500, Rob Crittenden wrote:
>>>> Christopher Young wrote:
>>>>> Ok.  I think I have a 'hint' here, but I could use some help getting
>> this fixed.
>>>>> 
>>>>> Comparing the two IPA servers, I found the following (modified SOME
>>>>> of the output myself):
>>>> 
>>>> You're right about the ipaCert. I'd export the renewed cert from your
>>>> working server using the same certutil command, just direct it to a
>>>> file instead. Then import it into the non-working server and restart
>>>> the httpd service. That should do it.
>>>> 
>>> I agree that this should fix it.
>>> 
>>> You could also try running `ipa-certupdate' as root, if the correct
>> certificate is
>>> to be found in cn=certificates,cn=ipa,cn=etc,{basedn}
>>> 
>>> Once everything is working again, you should check:
>>> 
>>> 1. renewal master configuration is correct
>>> 
>>> 2. certmonger tracking requests for the IPA RA cert are correct on
>>>   both servers
>>> 
>>> 3. the correct certificate is in
>>>   cn=certificates,cn=ipa,cn=etc,{basedn}
>>> 
>>> Thanks,
>>> Fraser
>>> 
>>>> Or you can try restarting certmonger on the non-working server to see
>>>> if
>>>> 
>>>> that triggers it to pull in the updated cert. Theoretically this
>>>> should do it as well but given potential past replication problems it
>>>> is possible the entry doesn't exist.
>>>> 
>>>> getcert list -n ipaCert on each will show some basic information. The
>>>> important thing is to see if there is some root cause that will make
>>>> this blow up again at renewal time.
>>>> 
>>>> rob
>>>> 
>>>>> 
>>>>> on 'ipa02' (the 'good' one):
>>>>> -
>>>>> ipa cert-show 1
>>>>>  Issuing CA: ipa
>>>>>  Certificate: <<>>
>>>>>  Subject: CN=Certificate Authority,O=.LOCAL
>>>>>  Issuer: CN=Certificate Authority,O=.L

Re: [Freeipa-users] Replica issue / Certificate Authority

[Florence Blanc-Renaud]

On 01/04/2017 12:41 PM, Christophe TREFOIS wrote:

Hi Fraser,

We encountered the same issue. We exported the certificate from a "good"
replica, using certutil.
We then used certutil -A -n ipaCert -d /etc/httpd/alias/ -i
/opt/sysadmin/cacert.crt -a -t CT,C on the bad server and then restarted
ipa, and certmonger.

Now, the certificate is correct both using the certutil command and the
getcert list -n ipaCert command.

However, we see an "ca-error: Invalid cookie: ''  "  in the output of
getcert list -n ipaCert.


Hi Christophe,

is this error displayed on the renewal master? If not, you can run
$ getcert resubmit -i 
and the error should go away. On non-renewal master, resubmit downloads 
the certificate from LDAP (it does not ask for renewal), meaning that 
this operation cannot be harmful.


To know which server is the renewal master:
$ ldapsearch -h localhost -p 389 -D 'cn=Directory Manager' -w password 
-b "cn=masters,cn=ipa,cn=etc,$BASEDN" 
'(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn

dn: cn=CA,cn=server.example.com,cn=masters,cn=ipa,cn=etc,$BASEDN

=> the renewal master is server.example.com

HTH,
Flo


Did we import the certificate correctly and should we worry about this
ca-error?
It seems replication is going fine, and also ipa-server-upgrade completes
successfully when run manually (whereas it failed previously from the same
error as in this thread)

Thanks for any pointers on how to clean the issue up properly,
Kind regards,

Christophe


-Original Message-
From: freeipa-users-boun...@redhat.com [mailto:freeipa-users-
boun...@redhat.com] On Behalf Of Fraser Tweedale
Sent: lundi 19 décembre 2016 00:11
To: Christopher Young 
Cc: freeipa-users@redhat.com
Subject: Re: [Freeipa-users] Replica issue / Certificate Authority

On Fri, Dec 16, 2016 at 05:29:18PM -0500, Rob Crittenden wrote:

Christopher Young wrote:

Ok.  I think I have a 'hint' here, but I could use some help getting

this fixed.


Comparing the two IPA servers, I found the following (modified SOME
of the output myself):


You're right about the ipaCert. I'd export the renewed cert from your
working server using the same certutil command, just direct it to a
file instead. Then import it into the non-working server and restart
the httpd service. That should do it.


I agree that this should fix it.

You could also try running `ipa-certupdate' as root, if the correct

certificate is

to be found in cn=certificates,cn=ipa,cn=etc,{basedn}

Once everything is working again, you should check:

1. renewal master configuration is correct

2. certmonger tracking requests for the IPA RA cert are correct on
   both servers

3. the correct certificate is in
   cn=certificates,cn=ipa,cn=etc,{basedn}

Thanks,
Fraser


Or you can try restarting certmonger on the non-working server to see
if

that triggers it to pull in the updated cert. Theoretically this
should do it as well but given potential past replication problems it
is possible the entry doesn't exist.

getcert list -n ipaCert on each will show some basic information. The
important thing is to see if there is some root cause that will make
this blow up again at renewal time.

rob



on 'ipa02' (the 'good' one):
-
ipa cert-show 1
  Issuing CA: ipa
  Certificate: <<>>
  Subject: CN=Certificate Authority,O=.LOCAL
  Issuer: CN=Certificate Authority,O=.LOCAL
  Not Before: Thu Jan 01 06:23:38 2015 UTC
  Not After: Mon Jan 01 06:23:38 2035 UTC
  Fingerprint (MD5): a6:aa:88:d4:66:e2:70:c1:e3:8c:37:0b:f3:eb:19:7d
  Fingerprint (SHA1):
11:c2:5a:58:bc:77:55:37:39:9b:13:b1:1a:a2:02:50:be:2e:a0:7f
  Serial number: 1
  Serial number (hex): 0x1
  Revoked: False
--


on 'ipa01'
-
ipa cert-show 1
ipa: ERROR: Certificate operation cannot be completed: EXCEPTION
(Invalid Credential.)
-

Thinking about this, I decided to just start checking for
inconsistencies and I noticed the following:

ipa02:
---
[root@ipa02 ~]# certutil -L -d /etc/httpd/alias/ -n ipaCert -a |
openssl x509 -text  | head
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 268304413 (0xffe001d)
Signature Algorithm: sha256WithRSAEncryption
Issuer: O=x.LOCAL, CN=Certificate Authority
Validity
Not Before: Nov 23 18:19:31 2016 GMT
Not After : Nov 13 18:19:31 2018 GMT
Subject: O=x.LOCAL, CN=IPA RA

--
ipa01:
--
[root@ipa01 tmp]# certutil -L -d /etc/httpd/alias/ -n ipaCert -a |
openssl x509 -text  | head
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 7 (0x7)
Signature Algorithm: sha256WithRSAEncryption
Issuer: O=.LOCAL, CN=Certificate Authority
Validity
Not Before: Jan  1 06:24:23 2015 GMT
Not After : Dec 21 06:24:23 2016 GMT
Subject: O=.LOCAL, CN=IPA RA

--

So, it looks like somewher

Re: [Freeipa-users] Replica issue / Certificate Authority

[Christophe TREFOIS]

Hi Fraser,

We encountered the same issue. We exported the certificate from a "good"
replica, using certutil.
We then used certutil -A -n ipaCert -d /etc/httpd/alias/ -i
/opt/sysadmin/cacert.crt -a -t CT,C on the bad server and then restarted
ipa, and certmonger.

Now, the certificate is correct both using the certutil command and the
getcert list -n ipaCert command.

However, we see an "ca-error: Invalid cookie: ''  "  in the output of
getcert list -n ipaCert.

Did we import the certificate correctly and should we worry about this
ca-error?
It seems replication is going fine, and also ipa-server-upgrade completes
successfully when run manually (whereas it failed previously from the same
error as in this thread)

Thanks for any pointers on how to clean the issue up properly,
Kind regards,

Christophe

> -Original Message-
> From: freeipa-users-boun...@redhat.com [mailto:freeipa-users-
> boun...@redhat.com] On Behalf Of Fraser Tweedale
> Sent: lundi 19 décembre 2016 00:11
> To: Christopher Young 
> Cc: freeipa-users@redhat.com
> Subject: Re: [Freeipa-users] Replica issue / Certificate Authority
> 
> On Fri, Dec 16, 2016 at 05:29:18PM -0500, Rob Crittenden wrote:
> > Christopher Young wrote:
> > > Ok.  I think I have a 'hint' here, but I could use some help getting
this fixed.
> > >
> > > Comparing the two IPA servers, I found the following (modified SOME
> > > of the output myself):
> >
> > You're right about the ipaCert. I'd export the renewed cert from your
> > working server using the same certutil command, just direct it to a
> > file instead. Then import it into the non-working server and restart
> > the httpd service. That should do it.
> >
> I agree that this should fix it.
> 
> You could also try running `ipa-certupdate' as root, if the correct
certificate is
> to be found in cn=certificates,cn=ipa,cn=etc,{basedn}
> 
> Once everything is working again, you should check:
> 
> 1. renewal master configuration is correct
> 
> 2. certmonger tracking requests for the IPA RA cert are correct on
>both servers
> 
> 3. the correct certificate is in
>cn=certificates,cn=ipa,cn=etc,{basedn}
> 
> Thanks,
> Fraser
> 
> > Or you can try restarting certmonger on the non-working server to see
> > if
> >
> > that triggers it to pull in the updated cert. Theoretically this
> > should do it as well but given potential past replication problems it
> > is possible the entry doesn't exist.
> >
> > getcert list -n ipaCert on each will show some basic information. The
> > important thing is to see if there is some root cause that will make
> > this blow up again at renewal time.
> >
> > rob
> >
> > >
> > > on 'ipa02' (the 'good' one):
> > > -
> > > ipa cert-show 1
> > >   Issuing CA: ipa
> > >   Certificate: <<>>
> > >   Subject: CN=Certificate Authority,O=.LOCAL
> > >   Issuer: CN=Certificate Authority,O=.LOCAL
> > >   Not Before: Thu Jan 01 06:23:38 2015 UTC
> > >   Not After: Mon Jan 01 06:23:38 2035 UTC
> > >   Fingerprint (MD5): a6:aa:88:d4:66:e2:70:c1:e3:8c:37:0b:f3:eb:19:7d
> > >   Fingerprint (SHA1):
> > > 11:c2:5a:58:bc:77:55:37:39:9b:13:b1:1a:a2:02:50:be:2e:a0:7f
> > >   Serial number: 1
> > >   Serial number (hex): 0x1
> > >   Revoked: False
> > > --
> > >
> > >
> > > on 'ipa01'
> > > -
> > > ipa cert-show 1
> > > ipa: ERROR: Certificate operation cannot be completed: EXCEPTION
> > > (Invalid Credential.)
> > > -
> > >
> > > Thinking about this, I decided to just start checking for
> > > inconsistencies and I noticed the following:
> > >
> > > ipa02:
> > > ---
> > > [root@ipa02 ~]# certutil -L -d /etc/httpd/alias/ -n ipaCert -a |
> > > openssl x509 -text  | head
> > > Certificate:
> > > Data:
> > > Version: 3 (0x2)
> > > Serial Number: 268304413 (0xffe001d)
> > > Signature Algorithm: sha256WithRSAEncryption
> > > Issuer: O=x.LOCAL, CN=Certificate Authority
> > > Validity
> > > Not Before: Nov 23 18:19:31 2016 GMT
> > > Not After : Nov 13 18:19:31 2018 GMT
> > > Subject: O=x.LOCAL, CN=IPA RA
> > >
> > > --
> > > ipa01:
> > > --
> > > [root@ipa01 tmp]# certutil -L -d /etc/httpd/alias/ -n ipaCert -a |
> > > op

Re: [Freeipa-users] Replica issue / Certificate Authority

[Fraser Tweedale]

On Fri, Dec 16, 2016 at 05:29:18PM -0500, Rob Crittenden wrote:
> Christopher Young wrote:
> > Ok.  I think I have a 'hint' here, but I could use some help getting this 
> > fixed.
> > 
> > Comparing the two IPA servers, I found the following (modified SOME of
> > the output myself):
> 
> You're right about the ipaCert. I'd export the renewed cert from your
> working server using the same certutil command, just direct it to a file
> instead. Then import it into the non-working server and restart the
> httpd service. That should do it.
> 
I agree that this should fix it.

You could also try running `ipa-certupdate' as root, if the correct
certificate is to be found in cn=certificates,cn=ipa,cn=etc,{basedn}

Once everything is working again, you should check:

1. renewal master configuration is correct

2. certmonger tracking requests for the IPA RA cert are correct on
   both servers

3. the correct certificate is in
   cn=certificates,cn=ipa,cn=etc,{basedn}

Thanks,
Fraser

> Or you can try restarting certmonger on the non-working server to see if
>
> that triggers it to pull in the updated cert. Theoretically this should
> do it as well but given potential past replication problems it is
> possible the entry doesn't exist.
> 
> getcert list -n ipaCert on each will show some basic information. The
> important thing is to see if there is some root cause that will make
> this blow up again at renewal time.
> 
> rob
> 
> > 
> > on 'ipa02' (the 'good' one):
> > -
> > ipa cert-show 1
> >   Issuing CA: ipa
> >   Certificate: <<>>
> >   Subject: CN=Certificate Authority,O=.LOCAL
> >   Issuer: CN=Certificate Authority,O=.LOCAL
> >   Not Before: Thu Jan 01 06:23:38 2015 UTC
> >   Not After: Mon Jan 01 06:23:38 2035 UTC
> >   Fingerprint (MD5): a6:aa:88:d4:66:e2:70:c1:e3:8c:37:0b:f3:eb:19:7d
> >   Fingerprint (SHA1):
> > 11:c2:5a:58:bc:77:55:37:39:9b:13:b1:1a:a2:02:50:be:2e:a0:7f
> >   Serial number: 1
> >   Serial number (hex): 0x1
> >   Revoked: False
> > --
> > 
> > 
> > on 'ipa01'
> > -
> > ipa cert-show 1
> > ipa: ERROR: Certificate operation cannot be completed: EXCEPTION
> > (Invalid Credential.)
> > -
> > 
> > Thinking about this, I decided to just start checking for
> > inconsistencies and I noticed the following:
> > 
> > ipa02:
> > ---
> > [root@ipa02 ~]# certutil -L -d /etc/httpd/alias/ -n ipaCert -a |
> > openssl x509 -text  | head
> > Certificate:
> > Data:
> > Version: 3 (0x2)
> > Serial Number: 268304413 (0xffe001d)
> > Signature Algorithm: sha256WithRSAEncryption
> > Issuer: O=x.LOCAL, CN=Certificate Authority
> > Validity
> > Not Before: Nov 23 18:19:31 2016 GMT
> > Not After : Nov 13 18:19:31 2018 GMT
> > Subject: O=x.LOCAL, CN=IPA RA
> > 
> > --
> > ipa01:
> > --
> > [root@ipa01 tmp]# certutil -L -d /etc/httpd/alias/ -n ipaCert -a |
> > openssl x509 -text  | head
> > Certificate:
> > Data:
> > Version: 3 (0x2)
> > Serial Number: 7 (0x7)
> > Signature Algorithm: sha256WithRSAEncryption
> > Issuer: O=.LOCAL, CN=Certificate Authority
> > Validity
> > Not Before: Jan  1 06:24:23 2015 GMT
> > Not After : Dec 21 06:24:23 2016 GMT
> > Subject: O=.LOCAL, CN=IPA RA
> > 
> > --
> > 
> > So, it looks like somewhere in the process, the certificate got
> > renewed but not updated on 'ipa01'?  I apologize as I'm trying to
> > understand this.  I believe that my end goal is probably still the
> > same (verify replication and get things working properly on the
> > 'ipa01' system.
> > 
> > Any help is very much appreciated!
> > 
> > -- Chris
> > 
> > 
> > On Fri, Dec 16, 2016 at 3:35 PM, Christopher Young
> >  wrote:
> >> I'm hoping to provide enough information to get some help to a very
> >> important issue that I'm currently having.
> >>
> >> I have two IPA servers at a single location that recently had a
> >> replication issue that I eventually resolved by reinitializing one of
> >> the masters/replicas with one that seemed to be the most 'good'.
> >>
> >> In any case, somewhere in this process, the new IPA 4.4 was release
> >> with/for CentOS 7.3.
> >>
> >> At this moment, regular replication seems to be working properly (in
> >> that I don't have any obvious issues and web interfaces on both
> >> systems seem to be consistent for updates EXCEPT when it comes to the
> >> certificates).
> >>
> >> Before I get to the errors, here is the output of some of the commands
> >> that I would expect anyone would need:
> >>
> >> --
> >> [root@ipa01 ~]# ipa-replica-manage list
> >> ipa01.passur.local: master
> >> ipa02.passur.local: master
> >> -
> >> [root@ipa01 ~]# ipa-replica-manage list -v ipa01.passur.local
> >> ipa02.passur.local: replica
> >>   last init status: None
> >>   last init ended: 1970-01-01 00:00:00+00:00
> >>   last update status: Error (0) Replica acquired successfully:
> >> I

Re: [Freeipa-users] Replica issue / Certificate Authority

[Rob Crittenden]

Christopher Young wrote:
> Ok.  I think I have a 'hint' here, but I could use some help getting this 
> fixed.
> 
> Comparing the two IPA servers, I found the following (modified SOME of
> the output myself):

You're right about the ipaCert. I'd export the renewed cert from your
working server using the same certutil command, just direct it to a file
instead. Then import it into the non-working server and restart the
httpd service. That should do it.

Or you can try restarting certmonger on the non-working server to see if
that triggers it to pull in the updated cert. Theoretically this should
do it as well but given potential past replication problems it is
possible the entry doesn't exist.

getcert list -n ipaCert on each will show some basic information. The
important thing is to see if there is some root cause that will make
this blow up again at renewal time.

rob

> 
> on 'ipa02' (the 'good' one):
> -
> ipa cert-show 1
>   Issuing CA: ipa
>   Certificate: <<>>
>   Subject: CN=Certificate Authority,O=.LOCAL
>   Issuer: CN=Certificate Authority,O=.LOCAL
>   Not Before: Thu Jan 01 06:23:38 2015 UTC
>   Not After: Mon Jan 01 06:23:38 2035 UTC
>   Fingerprint (MD5): a6:aa:88:d4:66:e2:70:c1:e3:8c:37:0b:f3:eb:19:7d
>   Fingerprint (SHA1):
> 11:c2:5a:58:bc:77:55:37:39:9b:13:b1:1a:a2:02:50:be:2e:a0:7f
>   Serial number: 1
>   Serial number (hex): 0x1
>   Revoked: False
> --
> 
> 
> on 'ipa01'
> -
> ipa cert-show 1
> ipa: ERROR: Certificate operation cannot be completed: EXCEPTION
> (Invalid Credential.)
> -
> 
> Thinking about this, I decided to just start checking for
> inconsistencies and I noticed the following:
> 
> ipa02:
> ---
> [root@ipa02 ~]# certutil -L -d /etc/httpd/alias/ -n ipaCert -a |
> openssl x509 -text  | head
> Certificate:
> Data:
> Version: 3 (0x2)
> Serial Number: 268304413 (0xffe001d)
> Signature Algorithm: sha256WithRSAEncryption
> Issuer: O=x.LOCAL, CN=Certificate Authority
> Validity
> Not Before: Nov 23 18:19:31 2016 GMT
> Not After : Nov 13 18:19:31 2018 GMT
> Subject: O=x.LOCAL, CN=IPA RA
> 
> --
> ipa01:
> --
> [root@ipa01 tmp]# certutil -L -d /etc/httpd/alias/ -n ipaCert -a |
> openssl x509 -text  | head
> Certificate:
> Data:
> Version: 3 (0x2)
> Serial Number: 7 (0x7)
> Signature Algorithm: sha256WithRSAEncryption
> Issuer: O=.LOCAL, CN=Certificate Authority
> Validity
> Not Before: Jan  1 06:24:23 2015 GMT
> Not After : Dec 21 06:24:23 2016 GMT
> Subject: O=.LOCAL, CN=IPA RA
> 
> --
> 
> So, it looks like somewhere in the process, the certificate got
> renewed but not updated on 'ipa01'?  I apologize as I'm trying to
> understand this.  I believe that my end goal is probably still the
> same (verify replication and get things working properly on the
> 'ipa01' system.
> 
> Any help is very much appreciated!
> 
> -- Chris
> 
> 
> On Fri, Dec 16, 2016 at 3:35 PM, Christopher Young
>  wrote:
>> I'm hoping to provide enough information to get some help to a very
>> important issue that I'm currently having.
>>
>> I have two IPA servers at a single location that recently had a
>> replication issue that I eventually resolved by reinitializing one of
>> the masters/replicas with one that seemed to be the most 'good'.
>>
>> In any case, somewhere in this process, the new IPA 4.4 was release
>> with/for CentOS 7.3.
>>
>> At this moment, regular replication seems to be working properly (in
>> that I don't have any obvious issues and web interfaces on both
>> systems seem to be consistent for updates EXCEPT when it comes to the
>> certificates).
>>
>> Before I get to the errors, here is the output of some of the commands
>> that I would expect anyone would need:
>>
>> --
>> [root@ipa01 ~]# ipa-replica-manage list
>> ipa01.passur.local: master
>> ipa02.passur.local: master
>> -
>> [root@ipa01 ~]# ipa-replica-manage list -v ipa01.passur.local
>> ipa02.passur.local: replica
>>   last init status: None
>>   last init ended: 1970-01-01 00:00:00+00:00
>>   last update status: Error (0) Replica acquired successfully:
>> Incremental update succeeded
>>   last update ended: 2016-12-16 20:25:40+00:00
>> -
>> [root@ipa01 ~]# ipa-replica-manage list -v ipa02.passur.local
>> ipa01.passur.local: replica
>>   last init status: None
>>   last init ended: 1970-01-01 00:00:00+00:00
>>   last update status: Error (0) Replica acquired successfully:
>> Incremental update succeeded
>>   last update ended: 2016-12-16 20:25:40+00:00
>> -
>> [root@ipa01 ~]# ipa-replica-manage list-ruv
>> Replica Update Vectors:
>> ipa01.passur.local:389: 4
>> ipa02.passur.local:389: 6
>> Certificate Server Replica Update Vectors:
>> ipa02.passur.local:389: 97
>> ipa01.passur.local:389: 96
>> --
>>
>>
>> After the yum updates were applied to each system, I not

Re: [Freeipa-users] Replica issue / Certificate Authority

[Christopher Young]

Ok.  I think I have a 'hint' here, but I could use some help getting this fixed.

Comparing the two IPA servers, I found the following (modified SOME of
the output myself):

on 'ipa02' (the 'good' one):
-
ipa cert-show 1
  Issuing CA: ipa
  Certificate: <<>>
  Subject: CN=Certificate Authority,O=.LOCAL
  Issuer: CN=Certificate Authority,O=.LOCAL
  Not Before: Thu Jan 01 06:23:38 2015 UTC
  Not After: Mon Jan 01 06:23:38 2035 UTC
  Fingerprint (MD5): a6:aa:88:d4:66:e2:70:c1:e3:8c:37:0b:f3:eb:19:7d
  Fingerprint (SHA1):
11:c2:5a:58:bc:77:55:37:39:9b:13:b1:1a:a2:02:50:be:2e:a0:7f
  Serial number: 1
  Serial number (hex): 0x1
  Revoked: False
--


on 'ipa01'
-
ipa cert-show 1
ipa: ERROR: Certificate operation cannot be completed: EXCEPTION
(Invalid Credential.)
-

Thinking about this, I decided to just start checking for
inconsistencies and I noticed the following:

ipa02:
---
[root@ipa02 ~]# certutil -L -d /etc/httpd/alias/ -n ipaCert -a |
openssl x509 -text  | head
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 268304413 (0xffe001d)
Signature Algorithm: sha256WithRSAEncryption
Issuer: O=x.LOCAL, CN=Certificate Authority
Validity
Not Before: Nov 23 18:19:31 2016 GMT
Not After : Nov 13 18:19:31 2018 GMT
Subject: O=x.LOCAL, CN=IPA RA

--
ipa01:
--
[root@ipa01 tmp]# certutil -L -d /etc/httpd/alias/ -n ipaCert -a |
openssl x509 -text  | head
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 7 (0x7)
Signature Algorithm: sha256WithRSAEncryption
Issuer: O=.LOCAL, CN=Certificate Authority
Validity
Not Before: Jan  1 06:24:23 2015 GMT
Not After : Dec 21 06:24:23 2016 GMT
Subject: O=.LOCAL, CN=IPA RA

--

So, it looks like somewhere in the process, the certificate got
renewed but not updated on 'ipa01'?  I apologize as I'm trying to
understand this.  I believe that my end goal is probably still the
same (verify replication and get things working properly on the
'ipa01' system.

Any help is very much appreciated!

-- Chris


On Fri, Dec 16, 2016 at 3:35 PM, Christopher Young
 wrote:
> I'm hoping to provide enough information to get some help to a very
> important issue that I'm currently having.
>
> I have two IPA servers at a single location that recently had a
> replication issue that I eventually resolved by reinitializing one of
> the masters/replicas with one that seemed to be the most 'good'.
>
> In any case, somewhere in this process, the new IPA 4.4 was release
> with/for CentOS 7.3.
>
> At this moment, regular replication seems to be working properly (in
> that I don't have any obvious issues and web interfaces on both
> systems seem to be consistent for updates EXCEPT when it comes to the
> certificates).
>
> Before I get to the errors, here is the output of some of the commands
> that I would expect anyone would need:
>
> --
> [root@ipa01 ~]# ipa-replica-manage list
> ipa01.passur.local: master
> ipa02.passur.local: master
> -
> [root@ipa01 ~]# ipa-replica-manage list -v ipa01.passur.local
> ipa02.passur.local: replica
>   last init status: None
>   last init ended: 1970-01-01 00:00:00+00:00
>   last update status: Error (0) Replica acquired successfully:
> Incremental update succeeded
>   last update ended: 2016-12-16 20:25:40+00:00
> -
> [root@ipa01 ~]# ipa-replica-manage list -v ipa02.passur.local
> ipa01.passur.local: replica
>   last init status: None
>   last init ended: 1970-01-01 00:00:00+00:00
>   last update status: Error (0) Replica acquired successfully:
> Incremental update succeeded
>   last update ended: 2016-12-16 20:25:40+00:00
> -
> [root@ipa01 ~]# ipa-replica-manage list-ruv
> Replica Update Vectors:
> ipa01.passur.local:389: 4
> ipa02.passur.local:389: 6
> Certificate Server Replica Update Vectors:
> ipa02.passur.local:389: 97
> ipa01.passur.local:389: 96
> --
>
>
> After the yum updates were applied to each system, I noticed that the
> results of 'ipa-server-upgrade' were quite different.  The 'ipa02'
> system went through without errors (this was also the system I used to
> reinitialize the other when I had a replication issue recently).
>
>
>
> On 'ipa01', I have following at the end of the 'ipaupgrade.log' file:
> --
> 2016-12-14T18:09:26Z ERROR IPA server upgrade failed: Inspect
> /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
> 2016-12-14T18:09:26Z DEBUG   File
> "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171,
> in execute
> return_value = self.run()
>   File 
> "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
> line 46, in run
> server.upgrade()
>   File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
> line 1863, in upgrade
> upgrade_configuration()
>   File "/usr/lib/python2.7/site-packages/ip

[Freeipa-users] Replica issue / Certificate Authority

[Christopher Young]

I'm hoping to provide enough information to get some help to a very
important issue that I'm currently having.

I have two IPA servers at a single location that recently had a
replication issue that I eventually resolved by reinitializing one of
the masters/replicas with one that seemed to be the most 'good'.

In any case, somewhere in this process, the new IPA 4.4 was release
with/for CentOS 7.3.

At this moment, regular replication seems to be working properly (in
that I don't have any obvious issues and web interfaces on both
systems seem to be consistent for updates EXCEPT when it comes to the
certificates).

Before I get to the errors, here is the output of some of the commands
that I would expect anyone would need:

--
[root@ipa01 ~]# ipa-replica-manage list
ipa01.passur.local: master
ipa02.passur.local: master
-
[root@ipa01 ~]# ipa-replica-manage list -v ipa01.passur.local
ipa02.passur.local: replica
  last init status: None
  last init ended: 1970-01-01 00:00:00+00:00
  last update status: Error (0) Replica acquired successfully:
Incremental update succeeded
  last update ended: 2016-12-16 20:25:40+00:00
-
[root@ipa01 ~]# ipa-replica-manage list -v ipa02.passur.local
ipa01.passur.local: replica
  last init status: None
  last init ended: 1970-01-01 00:00:00+00:00
  last update status: Error (0) Replica acquired successfully:
Incremental update succeeded
  last update ended: 2016-12-16 20:25:40+00:00
-
[root@ipa01 ~]# ipa-replica-manage list-ruv
Replica Update Vectors:
ipa01.passur.local:389: 4
ipa02.passur.local:389: 6
Certificate Server Replica Update Vectors:
ipa02.passur.local:389: 97
ipa01.passur.local:389: 96
--


After the yum updates were applied to each system, I noticed that the
results of 'ipa-server-upgrade' were quite different.  The 'ipa02'
system went through without errors (this was also the system I used to
reinitialize the other when I had a replication issue recently).



On 'ipa01', I have following at the end of the 'ipaupgrade.log' file:
--
2016-12-14T18:09:26Z ERROR IPA server upgrade failed: Inspect
/var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2016-12-14T18:09:26Z DEBUG   File
"/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171,
in execute
return_value = self.run()
  File 
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
line 46, in run
server.upgrade()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
line 1863, in upgrade
upgrade_configuration()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
line 1785, in upgrade_configuration
ca_enable_ldap_profile_subsystem(ca)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
line 336, in ca_enable_ldap_profile_subsystem
cainstance.migrate_profiles_to_ldap()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
line 1984, in migrate_profiles_to_ldap
_create_dogtag_profile(profile_id, profile_data, overwrite=False)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
line 1990, in _create_dogtag_profile
with api.Backend.ra_certprofile as profile_api:
  File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py",
line 2060, in __enter__
raise errors.RemoteRetrieveError(reason=_('Failed to authenticate
to CA REST API'))

2016-12-14T18:09:26Z DEBUG The ipa-server-upgrade command failed,
exception: RemoteRetrieveError: Failed to authenticate to CA REST API
2016-12-14T18:09:26Z ERROR Unexpected error - see
/var/log/ipaupgrade.log for details:
RemoteRetrieveError: Failed to authenticate to CA REST API
2016-12-14T18:09:26Z ERROR The ipa-server-upgrade command failed. See
/var/log/ipaupgrade.log for more information
--


In addition, when I go to the IPA web interface on the 'ipa01' system,
I get the following when I try to view any of the certificates:
--
IPA Error 4301: CertificateOperationError

Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)
--


I was wondering if there was a method for taking all the CA
details/tree/what have you from my 'ipa02' system and using it to
repopulate the 'ipa01'.   Since everything else seems to be working
correctly after a reinitialize on 'ipa01', I thought this would be the
safest way, but I'm opening any solutions as I need to get this fixed
ASAP.

Please let me know any additional details that may help OR if there is
a procedure that I could use to quickly and easily recreate 'ipa01'
WITH the certificate authority properly working on both.  I may need
some educate there.


Thanks!

-- Chris

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project