Marc Wiatrowski wrote:
Thanks for the reply Rob,

So should fixing replication be more than running a re-initialize?
I've tried this with no luck.  Still the same errors in renewing the IPA
certs.

re-init drops one database and replaces it with another. If you really did that then you have potentially lost a ton of records if indeed replication was stalled. Knowing what commands you ran would help to know for sure.

status: CA_UNREACHABLE
ca-error: Server at https://spider01a.iglass.net/ipa/xml failed request,
will retry: 4301 (RPC failed at server.  Certificate operation cannot be
completed: EXCEPTION (Certificate serial number 0x3ffe000f not found))

Is there a procedure for getting these serial numbers back in to the
system? or manually recreating somehow?

When IPA gets a certificate request and the host/service it is requesting it for already has a certificate, a revocation is done on the existing certificate (which in this case is failing because the cert is unknown). If you wipe out the usercertificate field from the entry ldap/spider01a.iglass.net then that should do it.


I was able to clear 4301 error.  One ipaCert needed to be updated.

Great!

rob


thanks

On Thu, Jun 16, 2016 at 10:22 AM, Rob Crittenden <rcrit...@redhat.com
<mailto:rcrit...@redhat.com>> wrote:

    Marc Wiatrowski wrote:

        Thanks Rob,

        Any suggestions on how make the CA aware of the current serial
        number?


    Serial numbers are dolled out like uid numbers, by the 389-ds DNA
    Plugin. So each CA that has ever issued a certificate has its own
    range, hence the quite different serial number values.

    Given that some issued certificates are unknown it stands to reason
    that replication is broken between one or more masters. Fixing that
    should resolve (most of) the other issues.

        Also started seeing the following error from two of the servers,
        spider01b and spider01o, but not spider01a when to navigate in
        the web
        gui.  Though it doesn't appear to stop me from doing anything.

        IPA Error 4301
        Certificate operation cannot be completed: EXCEPTION (Invalid
        Crential.)


    Dogtag does some of its access control by comparing the incoming
    client certificate with an expected value in its LDAP database, in
    this case uid=ipara,ou=People,o=ipaca. There you'll find a copy of
    the client certificate and a description field that contains the
    expected serial #, subject and issuer.

    These are out-of-whack if you're getting Invalid Credentials. It
    could be a number of things so I'd proceed cautiously. Given you
    have a working master I'd use that as a starting point.

    Look at the the RA cert is in /etc/httpd/alias:

    # certutil -L -d /etc/httpd/alias -n ipaCert | grep Serial

    See if it is the same on all masters, it should be.

    If it is, look at the uid=ipara entry on all the masters. Again,
    should be the same.

    Note that fixing this won't address any replication issues.

    rob


        Marc

        On Tue, Jun 14, 2016 at 2:07 PM, Marc Wiatrowski <w...@iglass.net
        <mailto:w...@iglass.net>
        <mailto:w...@iglass.net <mailto:w...@iglass.net>>> wrote:



             On Tue, Jun 14, 2016 at 11:22 AM, Rob Crittenden
             <rcrit...@redhat.com <mailto:rcrit...@redhat.com>
        <mailto:rcrit...@redhat.com <mailto:rcrit...@redhat.com>>> wrote:

                 Marc Wiatrowski wrote:

                     Hello, I'm having issues with the 3 ipa
        certificates of type
                     CA: IPA
                     renewing on 2 of 3 replicas.  Particularly on the 2
        that are
                     not the CA
                     master.  The other 5 certificates from getcert list
        do renew
                     and all
                     certificates on the CA master do look to renew.

                     Both servers running
                     ipa-server-3.0.0-50.el6.centos.1.x86_64  I've done
                     full updates and rebooted.


                 Can you check on the replication status for each CA?

                 $ ipa-csreplica-manage list -v ipa.example.com
        <http://ipa.example.com>
                 <http://ipa.example.com>

                 The hostname is important because including that will
        show the
                 agreements that host has. Do this for each master with
        a CA.

                 The CA being asked to do the renewal is unaware of the
        current
                 serial number so it is refusing to proceed.

                 rob



             [root@spider01o]$ ipa-csreplica-manage list -v
        spider01a.iglass.net <http://spider01a.iglass.net>
             <http://spider01a.iglass.net>
             Directory Manager password:

        spider01b.iglass.net <http://spider01b.iglass.net>
        <http://spider01b.iglass.net>
                last init status: None
                last init ended: None
                last update status: 0 Replica acquired successfully:
        Incremental
             update succeeded
                last update ended: 2016-06-14 17:49:16+00:00
        spider01o.iglass.net <http://spider01o.iglass.net>
        <http://spider01o.iglass.net>
                last init status: None
                last init ended: None
                last update status: 0 Replica acquired successfully:
        Incremental
             update started
                last update ended: 2016-06-14 17:55:20+00:00

             [root@spider01o]$ ipa-csreplica-manage list -v
        spider01o.iglass.net <http://spider01o.iglass.net>
             <http://spider01o.iglass.net>
             Directory Manager password:

        spider01a.iglass.net <http://spider01a.iglass.net>
        <http://spider01a.iglass.net>
                last init status: None
                last init ended: None
                last update status: 0 Replica acquired successfully:
        Incremental
             update started
                last update ended: 2016-06-14 17:57:44+00:00
        spider01b.iglass.net <http://spider01b.iglass.net>
        <http://spider01b.iglass.net>
                last init status: None
                last init ended: None
                last update status: 0 Replica acquired successfully:
        Incremental
             update started
                last update ended: 2016-06-14 17:57:41+00:00

             [root@spider01o]$ ipa-csreplica-manage list -v
        spider01b.iglass.net <http://spider01b.iglass.net>
             <http://spider01b.iglass.net>
             Directory Manager password:

        spider01a.iglass.net <http://spider01a.iglass.net>
        <http://spider01a.iglass.net>
                last init status: 0 Total update succeeded
                last init ended: 2016-06-03 19:43:12+00:00
                last update status: 0 Replica acquired successfully:
        Incremental
             update succeeded
                last update ended: 2016-06-14 17:44:17+00:00
        spider01o.iglass.net <http://spider01o.iglass.net>
        <http://spider01o.iglass.net>
                last init status: 0 Total update succeeded
                last init ended: 2016-06-03 19:44:38+00:00
                last update status: 0 Replica acquired successfully:
        Incremental
             update started
                last update ended: 2016-06-14 17:57:53+00:00
        spider01a.iglass.net <http://spider01a.iglass.net>
        <http://spider01a.iglass.net>
                last init status: None
                last init ended: None
                last update status: 0 Replica acquired successfully:
        Incremental
             update succeeded
                last update ended: 2016-06-14 17:44:13+00:00
        spider01o.iglass.net <http://spider01o.iglass.net>
        <http://spider01o.iglass.net>
                last init status: None
                last init ended: None
                last update status: 0 Replica acquired successfully:
        Incremental
             update started
                last update ended: 2016-06-14 17:57:54+00:00


             Not sure what this is telling... This an issue with the
        last being
             doubled?  Thanks



             The failed renews look like:

             [root@spider01a]$ getcert list -i 20141202144354
             Number of certificates and requests being tracked: 8.
             Request ID '20141202144354':
             status: CA_UNREACHABLE
             ca-error: Server at https://spider01a.iglass.net/ipa/xml
        failed request,
             will retry: 4301 (RPC failed at server.  Certificate
        operation cannot be
             completed: EXCEPTION (Certificate serial number 0x3ffe0010
        not found)).
             stuck: no
             key pair storage:

        
type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS
             Certificate DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile.txt'
             certificate:

        
type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS
             Certificate DB'
             CA: IPA
             issuer: CN=Certificate Authority,O=IGLASS.NET
        <http://IGLASS.NET>
             <http://iglass.net/> <http://IGLASS.NET <http://iglass.net/>>
             subject: CN=spider01a.iglass.net
        <http://spider01a.iglass.net> <http://spider01a.iglass.net/>
             <http://spider01a.iglass.net
             <http://spider01a.iglass.net/>>,O=IGLASS.NET
        <http://IGLASS.NET>
             <http://iglass.net/> <http://IGLASS.NET <http://iglass.net/>>
             expires: 2016-12-02 14:38:45 UTC
             key usage:

        digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
             eku: id-kp-serverAuth,id-kp-clientAuth
             pre-save command:
             post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv
        PKI-IPA
             track: yes
             auto-renew: yes

             [root@spider01a]$ getcert list -i 20141202144616
             Number of certificates and requests being tracked: 8.
             Request ID '20141202144616':
             status: CA_UNREACHABLE
             ca-error: Server at https://spider01a.iglass.net/ipa/xml
        failed request,
             will retry: 4301 (RPC failed at server.  Certificate
        operation cannot be
             completed: EXCEPTION (Certificate serial number 0x3ffe000f
        not found)).
             stuck: no
             key pair storage:

        
type=NSSDB,location='/etc/dirsrv/slapd-IGLASS-NET',nickname='Server-Cert',token='NSS
             Certificate
        DB',pinfile='/etc/dirsrv/slapd-IGLASS-NET/pwdfile.txt'
             certificate:

        
type=NSSDB,location='/etc/dirsrv/slapd-IGLASS-NET',nickname='Server-Cert',token='NSS
             Certificate DB'
             CA: IPA
             issuer: CN=Certificate Authority,O=IGLASS.NET
        <http://IGLASS.NET>
             <http://iglass.net/> <http://IGLASS.NET <http://iglass.net/>>
             subject: CN=spider01a.iglass.net
        <http://spider01a.iglass.net> <http://spider01a.iglass.net/>
             <http://spider01a.iglass.net
             <http://spider01a.iglass.net/>>,O=IGLASS.NET
        <http://IGLASS.NET>
             <http://iglass.net/> <http://IGLASS.NET <http://iglass.net/>>
             expires: 2016-12-02 14:38:43 UTC
             key usage:

        digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
             eku: id-kp-serverAuth,id-kp-clientAuth
             pre-save command:
             post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv
        IGLASS-NET
             track: yes
             auto-renew: yes

             [root@spider01a]$ getcert list -i 20141202144733
             Number of certificates and requests being tracked: 8.
             Request ID '20141202144733':
             status: CA_UNREACHABLE
             ca-error: Server at https://spider01a.iglass.net/ipa/xml
        failed request,
             will retry: 4301 (RPC failed at server.  Certificate
        operation cannot be
             completed: EXCEPTION (Certificate serial number 0x3ffe0011
        not found)).
             stuck: no
             key pair storage:

        type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
             Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
             certificate:

        type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
             Certificate DB'
             CA: IPA
             issuer: CN=Certificate Authority,O=IGLASS.NET
        <http://IGLASS.NET>
             <http://iglass.net/> <http://IGLASS.NET <http://iglass.net/>>
             subject: CN=spider01a.iglass.net
        <http://spider01a.iglass.net> <http://spider01a.iglass.net/>
             <http://spider01a.iglass.net
             <http://spider01a.iglass.net/>>,O=IGLASS.NET
        <http://IGLASS.NET>
             <http://iglass.net/> <http://IGLASS.NET <http://iglass.net/>>
             expires: 2016-12-02 14:38:46 UTC
             key usage:

        digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
             eku: id-kp-serverAuth,id-kp-clientAuth
             pre-save command:
             post-save command: /usr/lib64/ipa/certmonger/restart_httpd
             track: yes
             auto-renew: yes


             From
             [root@spider01a]$ getcert resubmit -i 20141202144354

             On the replica issuing the resubmit

             ==> /var/log/httpd/access_log <==
             192.168.176.2 - - [13/Jun/2016:15:49:32 -0400] "POST
        /ipa/xml HTTP/1.1"
             401 1370

             ==> /var/log/httpd/error_log <==
             [Mon Jun 13 15:49:33 2016] [error] ipa: ERROR:
             ipaserver.plugins.dogtag.ra.get_certificate(): EXCEPTION
        (Certificate
             serial number 0x3ffe0010 not found)
             [Mon Jun 13 15:49:33 2016] [error] ipa: INFO:
             host/spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>
             <mailto:spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>>
             <mailto:spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>
             <mailto:spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>>>:

        
cert_request(u'MIIDsTCCApkCAQAwNDETMBEGA1UEChMKSUdMQVNTLk5FVDEdMBsGA1UEAxMUc3BpZGVyMDFhLml...UVrN8lbKn17V5COjnj6k0mdbz3KptL0UI/l0BPlFBWGN5MFYaDx2F+y6LWv/aXeu2V4E6LA==',
             principal=u'dogtagldap/spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>
             <mailto:spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>>
             <mailto:spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>
             <mailto:spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>>>', add=True):
             CertificateOperationError

             ==> /var/log/httpd/access_log <==
             192.168.176.2 - - [13/Jun/2016:15:49:33 -0400] "POST
             /ca/agent/ca/displayBySerial HTTP/1.1" 200 262
             192.168.176.2 - host/spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>
             <mailto:spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>>
             <mailto:spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>
             <mailto:spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>>> [13/Jun/2016:15:49:32
        -0400]
             "POST /ipa/xml HTTP/1.1" 200 376

             ==> /var/log/pki-ca/system <==
             2508.TP-Processor6 - [13/Jun/2016:15:49:33 EDT] [3] [3] Servlet
             caDisplayBySerial: Error encountered in DisplayBySerial.
        Error Record
             not found.


             On the CA master spider01o:

             ==> /var/log/httpd/access_log <==
             192.168.176.2 - - [13/Jun/2016:15:49:33 -0400] "POST
        /ipa/xml HTTP/1.1"
             401 1370

             ==> krb5kdc.log <==
             Jun 13 15:49:34 spider01o.iglass.net
        <http://spider01o.iglass.net>
             <http://spider01o.iglass.net/> <http://spider01o.iglass.net
             <http://spider01o.iglass.net/>>
             krb5kdc[1963](info): TGS_REQ (4 etypes {18 17 16 23})
        192.168.177.2
             <http://192.168.177.2 <http://192.168.177.2/>>: ISSUE: authtime
             1465847372, etypes {rep=18
             tkt=18 ses=18}, host/spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>
             <mailto:spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>>
             <mailto:spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>
             <mailto:spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>>> for
             ldap/spider01o.iglass....@iglass.net
        <mailto:spider01o.iglass....@iglass.net>
             <mailto:spider01o.iglass....@iglass.net
        <mailto:spider01o.iglass....@iglass.net>>
             <mailto:spider01o.iglass....@iglass.net
        <mailto:spider01o.iglass....@iglass.net>
             <mailto:spider01o.iglass....@iglass.net
        <mailto:spider01o.iglass....@iglass.net>>>

             ==> /var/log/httpd/error_log <==
             [Mon Jun 13 15:49:34 2016] [error] ipa: ERROR:
             ipaserver.plugins.dogtag.ra.get_certificate(): EXCEPTION
        (Invalid
             Credential.)
             [Mon Jun 13 15:49:34 2016] [error] ipa: INFO:
             host/spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>
             <mailto:spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>>
             <mailto:spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>
             <mailto:spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>>>:

        
cert_request(u'MIIDsTCCApkCAQAwNDETMBEGA1UEChMKSUdMQVNTLk5FVDEdMBsGA1UEAxMUc3BpZGVyMDFhLml...UVrN8lbKn17V5COjnj6k0mdbz3KptL0UI/l0BPlFBWGN5MFYaDx2F+y6LWv/aXeu2V4E6LA==',
             principal=u'dogtagldap/spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>
             <mailto:spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>>
             <mailto:spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>
             <mailto:spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>>>', add=True):
             CertificateOperationError

             ==> /var/log/httpd/access_log <==
             192.168.177.2 - - [13/Jun/2016:15:49:34 -0400] "POST
             /ca/agent/ca/displayBySerial HTTP/1.1" 200 235
             192.168.176.2 - host/spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>
             <mailto:spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>>
             <mailto:spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>
             <mailto:spider01a.iglass....@iglass.net
        <mailto:spider01a.iglass....@iglass.net>>> [13/Jun/2016:15:49:33
        -0400]
             "POST /ipa/xml HTTP/1.1" 200 349

             ==> /var/log/pki-ca/system <==
             2231.TP-Processor3 - [13/Jun/2016:15:49:34 EDT] [6] [3] Cannot
             authenticate agent with certificate Serial 0x5ffc0008
        Subject DN CN=IPA
             RA,O=IGLASS.NET <http://IGLASS.NET> <http://iglass.net/>
        <http://IGLASS.NET
             <http://iglass.net/>>. Error: User not found


             I realize they expire at the end of the year, but I've had my
             certificates expire before and would rather not go through
        that again.
             Any idea on what's wrong or suggestions on where to look
        would be
             appreciated.

             Thanks,
             Marc







--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Reply via email to