Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

[Thorsten Scherf]

On [Tue, 29.03.2016 20:53], Timothy Geier wrote:



On Mar 29, 2016, at 2:00 AM, Thorsten Scherf  wrote:

On [Mon, 28.03.2016 18:18], Timothy Geier wrote:



On Mar 28, 2016, at 12:53 PM, Thorsten Scherf  wrote:

On [Sat, 26.03.2016 03:26], Timothy Geier wrote:

To follow up on this issue, we haven’t been able to get any further since
last month due to the missing caServerCert profile..the configuration
files /usr/share/pki/ca/profiles/ca/caServerCert.cfg
and /var/lib/pki/pki-tomcat/ca/profiles/ca/caServerCert.cfg are present
and are identical.   The pki-ca package
passes rpm -V as well.   Are there any other troubleshooting steps we can
take?


Can you please check if the profile is available in the LDAP trees:

# ldapsearch -LLLx -D "cn=Directory Manager" -W -b cn=certprofiles,cn=ca,$suffix


dn: cn=certprofiles,cn=ca,$suffix
objectClass: nsContainer
objectClass: top
cn: certprofiles


# ldapsearch -LLLx -D "cn=Directory Manager" -W -b 
ou=certificateProfiles,ou=ca,o=ipaca


dn: ou=certificateProfiles,ou=ca,o=ipaca
objectClass: top
objectClass: organizationalUnit
ou: certificateProfiles



If this is the case, please check if the profile is accessable by the
host:

# kinit -kt /etc/krb5.keytab; klist; ipa certprofile-show caIPAserviceCert



ipa: ERROR: caIPAserviceCert: Certificate Profile not found


I either suspect that the profiles have not been properly migrated to
the LDAP tree or that some ACIs are missing to allow access to the
profiles.



I suspect you’re right..I ran these same commands on a reference system and 
there was
a lot more output in the ldapsearches and the ipa certprofile-show command came 
back with
Profile ID: caIPAserviceCert
Profile description: Standard profile for network services
Store issued certificates: TRUE


Yes, this is a known issue which has been fixed in the most recent
FreeIPA releases 4.2.4 and 4.3.1.
I would recommend to upgrade your system to one of those releases. If this is 
not feasible, I can send you instructions how to fix the issue manually.



It’s currently at 4.2.0-15.el7.centos.3..would the update 4.2.0-15.0.1.el7.centos.6 have the fix backported?  


The CentOS and Red Hat updates won't be released before May. The FreeIPA
updates are already available:

http://www.freeipa.org/page/Releases/4.2.4
http://www.freeipa.org/page/Releases/4.3.1


Also, should com.netscape.cmscore.profile be changed in 
/var/lib/pki/pki-tomcat/ca/conf/CS.cfg beforehand?


This is only necessary if you want to fix it manually. You don't need to
change it when you apply the updated packages.

Cheers,
Thorsten



Thanks,


Cheers,
Thorsten







"This message and any attachments may contain confidential information. If you
have received this  message in error, any use or distribution is prohibited.
Please notify us by reply e-mail if you have mistakenly received this message,
and immediately and permanently delete it and any attachments. Thank you."


--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

[Timothy Geier]

> On Mar 29, 2016, at 2:00 AM, Thorsten Scherf  wrote:
> 
> On [Mon, 28.03.2016 18:18], Timothy Geier wrote:
>> 
>>> On Mar 28, 2016, at 12:53 PM, Thorsten Scherf  wrote:
>>> 
>>> On [Sat, 26.03.2016 03:26], Timothy Geier wrote:
 To follow up on this issue, we haven’t been able to get any further since
 last month due to the missing caServerCert profile..the configuration
 files /usr/share/pki/ca/profiles/ca/caServerCert.cfg
 and /var/lib/pki/pki-tomcat/ca/profiles/ca/caServerCert.cfg are present
 and are identical.   The pki-ca package
 passes rpm -V as well.   Are there any other troubleshooting steps we can
 take?
>>> 
>>> Can you please check if the profile is available in the LDAP trees:
>>> 
>>> # ldapsearch -LLLx -D "cn=Directory Manager" -W -b 
>>> cn=certprofiles,cn=ca,$suffix
>> 
>> dn: cn=certprofiles,cn=ca,$suffix
>> objectClass: nsContainer
>> objectClass: top
>> cn: certprofiles
>> 
>>> # ldapsearch -LLLx -D "cn=Directory Manager" -W -b 
>>> ou=certificateProfiles,ou=ca,o=ipaca
>> 
>> dn: ou=certificateProfiles,ou=ca,o=ipaca
>> objectClass: top
>> objectClass: organizationalUnit
>> ou: certificateProfiles
>> 
>>> 
>>> If this is the case, please check if the profile is accessable by the
>>> host:
>>> 
>>> # kinit -kt /etc/krb5.keytab; klist; ipa certprofile-show caIPAserviceCert
>>> 
>> 
>> ipa: ERROR: caIPAserviceCert: Certificate Profile not found
>> 
>>> I either suspect that the profiles have not been properly migrated to
>>> the LDAP tree or that some ACIs are missing to allow access to the
>>> profiles.
>>> 
>> 
>> I suspect you’re right..I ran these same commands on a reference system and 
>> there was
>> a lot more output in the ldapsearches and the ipa certprofile-show command 
>> came back with
>> Profile ID: caIPAserviceCert
>> Profile description: Standard profile for network services
>> Store issued certificates: TRUE
> 
> Yes, this is a known issue which has been fixed in the most recent
> FreeIPA releases 4.2.4 and 4.3.1. 
> I would recommend to upgrade your system to one of those releases. If this is 
> not feasible, I can send you instructions how to fix the issue manually.
> 

It’s currently at 4.2.0-15.el7.centos.3..would the update 
4.2.0-15.0.1.el7.centos.6 have the fix backported?  Also, should 
com.netscape.cmscore.profile be changed in 
/var/lib/pki/pki-tomcat/ca/conf/CS.cfg beforehand?

Thanks,

> Cheers,
> Thorsten
> 





"This message and any attachments may contain confidential information. If you
have received this  message in error, any use or distribution is prohibited. 
Please notify us by reply e-mail if you have mistakenly received this message,
and immediately and permanently delete it and any attachments. Thank you."

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

[Thorsten Scherf]

On [Mon, 28.03.2016 18:18], Timothy Geier wrote:



On Mar 28, 2016, at 12:53 PM, Thorsten Scherf  wrote:

On [Sat, 26.03.2016 03:26], Timothy Geier wrote:

 To follow up on this issue, we haven’t been able to get any further since
 last month due to the missing caServerCert profile..the configuration
 files /usr/share/pki/ca/profiles/ca/caServerCert.cfg
 and /var/lib/pki/pki-tomcat/ca/profiles/ca/caServerCert.cfg are present
 and are identical.   The pki-ca package
 passes rpm -V as well.   Are there any other troubleshooting steps we can
 take?


Can you please check if the profile is available in the LDAP trees:

# ldapsearch -LLLx -D "cn=Directory Manager" -W -b cn=certprofiles,cn=ca,$suffix


dn: cn=certprofiles,cn=ca,$suffix
objectClass: nsContainer
objectClass: top
cn: certprofiles


# ldapsearch -LLLx -D "cn=Directory Manager" -W -b 
ou=certificateProfiles,ou=ca,o=ipaca


dn: ou=certificateProfiles,ou=ca,o=ipaca
objectClass: top
objectClass: organizationalUnit
ou: certificateProfiles



If this is the case, please check if the profile is accessable by the
host:

# kinit -kt /etc/krb5.keytab; klist; ipa certprofile-show caIPAserviceCert



ipa: ERROR: caIPAserviceCert: Certificate Profile not found


I either suspect that the profiles have not been properly migrated to
the LDAP tree or that some ACIs are missing to allow access to the
profiles.



I suspect you’re right..I ran these same commands on a reference system and 
there was
a lot more output in the ldapsearches and the ipa certprofile-show command came 
back with
 Profile ID: caIPAserviceCert
 Profile description: Standard profile for network services
 Store issued certificates: TRUE


Yes, this is a known issue which has been fixed in the most recent
FreeIPA releases 4.2.4 and 4.3.1. 

I would recommend to upgrade your system to one of those releases. If this 
is not feasible, I can send you instructions how to fix the issue manually.


Cheers,
Thorsten

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

[Fraser Tweedale]

On Mon, Mar 28, 2016 at 10:55:06AM -0500, Endi Sukma Dewata wrote:
> On 3/28/2016 10:00 AM, Rob Crittenden wrote:
> >Timothy Geier wrote:
> >>>Thanks for the procedure..the good news is this worked quite
> >>>well in making sure that 389 didn’t crash immediately after
> >>>startup.  The bad news is that the certificates still didn’t
> >>>renew due to
> >>>
> >>>Server at "http://master_server:8080/ca/ee/ca/profileSubmit
> >>>"
> >>>
> >>>replied: Profile caServerCert Not Found
> >>>
> >>>which was the same error in getcert list I saw that one time
> >>>389 didn’t crash right away.  At least now this can be further
> >>>troubleshooted without worrying about 389.
> >>>
> >>>
> >>
> >>To follow up on this issue, we haven’t been able to get any
> >>further since last month due to the missing caServerCert
> >>profile..the configuration files
> >>/usr/share/pki/ca/profiles/ca/caServerCert.cfg and
> >>/var/lib/pki/pki-tomcat/ca/profiles/ca/caServerCert.cfg are
> >>present and are identical.   The pki-ca package passes rpm -V as
> >>well.   Are there any other troubleshooting steps we can take?
> >
> >Maybe Endi or Ade have some ideas why the CA isn't recognizing
> >the profile.
> >
> >rob
> >
> 
> Fraser, is it possible the profile is missing from LDAP?
> 
There is a ticket for a situation where migration of profiles to
LDAP does not occur:
https://bugzilla.redhat.com/show_bug.cgi?id=1300252

See also upstream ticket:
https://fedorahosted.org/freeipa/ticket/5682

The fix is awaiting release for RHEL.

A possible workaround is to modify
/var/lib/pki/pki-tomcat/ca/conf/CS.cfg, replacing the value:

com.netscape.cmscore.profile.LDAPProfileSubsystem

with:

com.netscape.cmscore.profile.ProfileSubsystem

Then running `ipa-server-upgrade`.  The upgrade program should
observe that LDAP-based profiles are not enabled, re-enable the
LDAPProfileSubsystem and import all file-based profiles into the
database.

If you are able to try this procedure, let me know how it goes.

Cheers,
Fraser

> Timothy, could you provide us with the CA debug logs
> (/var/log/pki/pki-tomcat/ca/debug) and CA configuration file
> (/var/lib/pki/pki-tomcat/ca/conf/CS.cfg)?
> 
> Thanks!
> 
> -- 
> Endi S. Dewata

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

[Timothy Geier]

> On Mar 28, 2016, at 12:53 PM, Thorsten Scherf  wrote:
> 
> On [Sat, 26.03.2016 03:26], Timothy Geier wrote:
>>  To follow up on this issue, we haven’t been able to get any further since
>>  last month due to the missing caServerCert profile..the configuration
>>  files /usr/share/pki/ca/profiles/ca/caServerCert.cfg
>>  and /var/lib/pki/pki-tomcat/ca/profiles/ca/caServerCert.cfg are present
>>  and are identical.   The pki-ca package
>>  passes rpm -V as well.   Are there any other troubleshooting steps we can
>>  take?
> 
> Can you please check if the profile is available in the LDAP trees:
> 
> # ldapsearch -LLLx -D "cn=Directory Manager" -W -b 
> cn=certprofiles,cn=ca,$suffix

dn: cn=certprofiles,cn=ca,$suffix
objectClass: nsContainer
objectClass: top
cn: certprofiles

> # ldapsearch -LLLx -D "cn=Directory Manager" -W -b 
> ou=certificateProfiles,ou=ca,o=ipaca

dn: ou=certificateProfiles,ou=ca,o=ipaca
objectClass: top
objectClass: organizationalUnit
ou: certificateProfiles

> 
> If this is the case, please check if the profile is accessable by the
> host:
> 
> # kinit -kt /etc/krb5.keytab; klist; ipa certprofile-show caIPAserviceCert
> 

ipa: ERROR: caIPAserviceCert: Certificate Profile not found

> I either suspect that the profiles have not been properly migrated to
> the LDAP tree or that some ACIs are missing to allow access to the
> profiles.
> 

I suspect you’re right..I ran these same commands on a reference system and 
there was
a lot more output in the ldapsearches and the ipa certprofile-show command came 
back with
  Profile ID: caIPAserviceCert
  Profile description: Standard profile for network services
  Store issued certificates: TRUE

Thanks,

> Cheers,
> Thorsten
> 
> -- 
> Manage your subscription for the Freeipa-users mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go to http://freeipa.org for more info on the project





"This message and any attachments may contain confidential information. If you
have received this  message in error, any use or distribution is prohibited. 
Please notify us by reply e-mail if you have mistakenly received this message,
and immediately and permanently delete it and any attachments. Thank you."

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

[Rob Crittenden]

Timothy Geier wrote:



On Feb 28, 2016, at 2:15 AM, Timothy Geier > wrote:



On Feb 23, 2016, at 4:22 AM, Ludwig Krispenz > wrote:


On 02/22/2016 11:51 PM, Timothy Geier wrote:


What’s the established procedure to start a 389 instance without any
replication agreements enabled?  The only thing that seemed close on
google
(http://directory.fedoraproject.org/docs/389ds/howto/howto-fix-and-reset-time-skew.html)
seems risky and couldn’t be done
trivially in a production environment.

no, this is about how to get out of problems when replication could
no longer synchronize its csn time generation, either by too many
accumulate time drifts o playing with system time, hope you don't
have to go thru this.

Enabling disabling a replication agreement can be done by setting the
configuration parameter:

look for replication agreements (entries with
objectclass=nsDS5ReplicationAgreement) and set
nsds5ReplicaEnabled: off

you can do this with an ldapmodify when the server is running or by
editing /etc/dirsrv/slapd-/dse.ldif when teh server is stopped


Thanks for the procedure..the good news is this worked quite well in
making sure that 389 didn’t crash immediately after startup.  The bad
news is that the certificates still didn’t renew due to

Server at "http://master_server:8080/ca/ee/ca/profileSubmit
"
replied: Profile caServerCert Not Found

which was the same error in getcert list I saw that one time 389
didn’t crash right away.  At least now this can be further
troubleshooted without worrying about 389.




To follow up on this issue, we haven’t been able to get any further
since last month due to the missing caServerCert profile..the
configuration files /usr/share/pki/ca/profiles/ca/caServerCert.cfg
and /var/lib/pki/pki-tomcat/ca/profiles/ca/caServerCert.cfg are present
and are identical.   The pki-ca package
passes rpm -V as well.   Are there any other troubleshooting steps we
can take?


Maybe Endi or Ade have some ideas why the CA isn't recognizing the profile.

rob

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

[Timothy Geier]

On Feb 28, 2016, at 2:15 AM, Timothy Geier 
> wrote:


On Feb 23, 2016, at 4:22 AM, Ludwig Krispenz 
> wrote:


On 02/22/2016 11:51 PM, Timothy Geier wrote:

What’s the established procedure to start a 389 instance without any 
replication agreements enabled?  The only thing that seemed close on google 
(http://directory.fedoraproject.org/docs/389ds/howto/howto-fix-and-reset-time-skew.html)
 seems risky and couldn’t be done
trivially in a production environment.
no, this is about how to get out of problems when replication could no longer 
synchronize its csn time generation, either by too many accumulate time drifts 
o playing with system time, hope you don't have to go thru this.

Enabling disabling a replication agreement can be done by setting the 
configuration parameter:

look for replication agreements (entries with 
objectclass=nsDS5ReplicationAgreement) and set
nsds5ReplicaEnabled: off

you can do this with an ldapmodify when the server is running or by editing 
/etc/dirsrv/slapd-/dse.ldif when teh server is stopped

Thanks for the procedure..the good news is this worked quite well in making 
sure that 389 didn’t crash immediately after startup.  The bad news is that the 
certificates still didn’t renew due to

Server at 
"http://master_server:8080/ca/ee/ca/profileSubmit"
 replied: Profile caServerCert Not Found

which was the same error in getcert list I saw that one time 389 didn’t crash 
right away.  At least now this can be further troubleshooted without worrying 
about 389.



To follow up on this issue, we haven’t been able to get any further since last 
month due to the missing caServerCert profile..the configuration files 
/usr/share/pki/ca/profiles/ca/caServerCert.cfg and 
/var/lib/pki/pki-tomcat/ca/profiles/ca/caServerCert.cfg are present and are 
identical.   The pki-ca package
passes rpm -V as well.   Are there any other troubleshooting steps we can take?








"This message and any attachments may contain confidential information. If you
have received this  message in error, any use or distribution is prohibited. 
Please notify us by reply e-mail if you have mistakenly received this message,
and immediately and permanently delete it and any attachments. Thank you."-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

[Rob Crittenden]

Ludwig Krispenz wrote:
> The crash is an abort because of a failed assertion in  the kerberos code
> 
> Thread 1 (Thread 0x7fa7d4c88700 (LWP 3125)):
> #0  0x7fa7e6ace5f7 in raise () from /lib64/libc.so.6
> No symbol table info available.
> #1  0x7fa7e6acfce8 in abort () from /lib64/libc.so.6
> No symbol table info available.
> #2  0x7fa7e6ac7566 in __assert_fail_base () from /lib64/libc.so.6
> No symbol table info available.
> #3  0x7fa7e6ac7612 in __assert_fail () from /lib64/libc.so.6
> No symbol table info available.
> #4  0x7fa7e8d71b83 in k5_mutex_lock.part.1 () from /lib64/libkrb5.so.3
> No symbol table info available.
> #5  0x7fa7e8d7bda1 in k5_cc_mutex_lock () from /lib64/libkrb5.so.3
> No symbol table info available.
> #6  0x7fa7e8d851bf in krb5_mcc_store () from /lib64/libkrb5.so.3
> No symbol table info available.
> #7  0x7fa7e8d88070 in krb5_cc_store_cred () from /lib64/libkrb5.so.3
> No symbol table info available.
> #8  0x7fa7e8d98be3 in complete () from /lib64/libkrb5.so.3
> No symbol table info available.
> #9  0x7fa7e8d99ba1 in krb5_tkt_creds_step () from /lib64/libkrb5.so.3
> No symbol table info available.
> #10 0x7fa7e8d9a637 in krb5_tkt_creds_get () from /lib64/libkrb5.so.3
> No symbol table info available.
> #11 0x7fa7e8d9a75c in krb5_get_credentials () from /lib64/libkrb5.so.3
> No symbol table info available.
> #12 0x7fa7e0f6736a in krb5_gss_init_sec_context_ext () from
> /lib64/libgssapi_krb5.so.2
> No symbol table info available.
> #13 0x7fa7e0f67c97 in krb5_gss_init_sec_context () from
> /lib64/libgssapi_krb5.so.2
> No symbol table info available.
> #14 0x7fa7e0f516ab in gss_init_sec_context () from
> /lib64/libgssapi_krb5.so.2
> No symbol table info available.
> #15 0x7fa7e118ac44 in gssapi_client_mech_step () from
> /usr/lib64/sasl2/libgssapiv2.so
> No symbol table info available.
> #16 0x7fa7e72847f5 in sasl_client_step () from /lib64/libsasl2.so.3
> No symbol table info available.
> #17 0x7fa7e7284b76 in sasl_client_start () from /lib64/libsasl2.so.3
> No symbol table info available.
> #18 0x7fa7e8475e73 in ldap_int_sasl_bind () from
> /lib64/libldap_r-2.4.so.2
> No symbol table info available.
> #19 0x7fa7e8479492 in ldap_sasl_interactive_bind () from
> /lib64/libldap_r-2.4.so.2
> No symbol table info available.
> #20 0x7fa7e84796bd in ldap_sasl_interactive_bind_s () from
> /lib64/libldap_r-2.4.so.2
> 
> Directory server tries to open a replication connetion usig GSSAPI. I
> don't know which assertion fails in krb, but
> - could you try with the replication agreement disabled ?
> - Rob, you have been discussing renewals of keytabs, would we have to
> renew the ds.keytab ?

This is in the context of renewing certificates, not keytabs.

I guess installing the krb5 debuginfo might give more details on where
it is crashing.

rob

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

[Ludwig Krispenz]

On 02/22/2016 11:51 PM, Timothy Geier wrote:


What’s the established procedure to start a 389 instance without any 
replication agreements enabled?  The only thing that seemed close on 
google 
(http://directory.fedoraproject.org/docs/389ds/howto/howto-fix-and-reset-time-skew.html) 
seems risky and couldn’t be done

trivially in a production environment.
no, this is about how to get out of problems when replication could no 
longer synchronize its csn time generation, either by too many 
accumulate time drifts o playing with system time, hope you don't have 
to go thru this.


Enabling disabling a replication agreement can be done by setting the 
configuration parameter:


look for replication agreements (entries with 
objectclass=nsDS5ReplicationAgreement) and set

nsds5ReplicaEnabled: off

you can do this with an ldapmodify when the server is running or by 
editing /etc/dirsrv/slapd-/dse.ldif when teh server is stopped


--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

[Timothy Geier]

On Feb 22, 2016, at 9:21 AM, Ludwig Krispenz 
> wrote:

The crash is an abort because of a failed assertion in  the kerberos code

Thread 1 (Thread 0x7fa7d4c88700 (LWP 3125)):
#0  0x7fa7e6ace5f7 in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x7fa7e6acfce8 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x7fa7e6ac7566 in __assert_fail_base () from /lib64/libc.so.6
No symbol table info available.
#3  0x7fa7e6ac7612 in __assert_fail () from /lib64/libc.so.6
No symbol table info available.
#4  0x7fa7e8d71b83 in k5_mutex_lock.part.1 () from /lib64/libkrb5.so.3
No symbol table info available.
#5  0x7fa7e8d7bda1 in k5_cc_mutex_lock () from /lib64/libkrb5.so.3
No symbol table info available.
#6  0x7fa7e8d851bf in krb5_mcc_store () from /lib64/libkrb5.so.3
No symbol table info available.
#7  0x7fa7e8d88070 in krb5_cc_store_cred () from /lib64/libkrb5.so.3
No symbol table info available.
#8  0x7fa7e8d98be3 in complete () from /lib64/libkrb5.so.3
No symbol table info available.
#9  0x7fa7e8d99ba1 in krb5_tkt_creds_step () from /lib64/libkrb5.so.3
No symbol table info available.
#10 0x7fa7e8d9a637 in krb5_tkt_creds_get () from /lib64/libkrb5.so.3
No symbol table info available.
#11 0x7fa7e8d9a75c in krb5_get_credentials () from /lib64/libkrb5.so.3
No symbol table info available.
#12 0x7fa7e0f6736a in krb5_gss_init_sec_context_ext () from 
/lib64/libgssapi_krb5.so.2
No symbol table info available.
#13 0x7fa7e0f67c97 in krb5_gss_init_sec_context () from 
/lib64/libgssapi_krb5.so.2
No symbol table info available.
#14 0x7fa7e0f516ab in gss_init_sec_context () from 
/lib64/libgssapi_krb5.so.2
No symbol table info available.
#15 0x7fa7e118ac44 in gssapi_client_mech_step () from 
/usr/lib64/sasl2/libgssapiv2.so
No symbol table info available.
#16 0x7fa7e72847f5 in sasl_client_step () from /lib64/libsasl2.so.3
No symbol table info available.
#17 0x7fa7e7284b76 in sasl_client_start () from /lib64/libsasl2.so.3
No symbol table info available.
#18 0x7fa7e8475e73 in ldap_int_sasl_bind () from /lib64/libldap_r-2.4.so.2
No symbol table info available.
#19 0x7fa7e8479492 in ldap_sasl_interactive_bind () from 
/lib64/libldap_r-2.4.so.2
No symbol table info available.
#20 0x7fa7e84796bd in ldap_sasl_interactive_bind_s () from 
/lib64/libldap_r-2.4.so.2

Directory server tries to open a replication connetion usig GSSAPI. I don't 
know which assertion fails in krb, but
- could you try with the replication agreement disabled ?
- Rob, you have been discussing renewals of keytabs, would we have to renew the 
ds.keytab ?


What’s the established procedure to start a 389 instance without any 
replication agreements enabled?  The only thing that seemed close on google 
(http://directory.fedoraproject.org/docs/389ds/howto/howto-fix-and-reset-time-skew.html)
 seems risky and couldn’t be done
trivially in a production environment.

Thanks,


On 02/20/2016 07:57 AM, Timothy Geier wrote:

To update this thread..timeshifting to before the certs expired (Feb.
2) brings up the interesting issue of pki-tomcatd (sometimes) starting
while slapd crashes almost immediately after ipactl start is run..if it
doesn't crash right away, pki-tomcatd will start, but since slapd will
then go down right afterwards, the certs cannot be renewed.  After
returning to real time, slapd is just fine along with everything else
aside from pki-tomcatd.

A core dump is attached.


"This message and any attachments may contain confidential information. If you
have received this  message in error, any use or distribution is prohibited.
Please notify us by reply e-mail if you have mistakenly received this message,
and immediately and permanently delete it and any attachments. Thank you."



--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project





"This message and any attachments may contain confidential information. If you
have received this  message in error, any use or distribution is prohibited. 
Please notify us by reply e-mail if you have mistakenly received this message,
and immediately and permanently delete it and any attachments. Thank you."-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

[Ludwig Krispenz]

The crash is an abort because of a failed assertion in  the kerberos code

Thread 1 (Thread 0x7fa7d4c88700 (LWP 3125)):
#0  0x7fa7e6ace5f7 in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x7fa7e6acfce8 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x7fa7e6ac7566 in __assert_fail_base () from /lib64/libc.so.6
No symbol table info available.
#3  0x7fa7e6ac7612 in __assert_fail () from /lib64/libc.so.6
No symbol table info available.
#4  0x7fa7e8d71b83 in k5_mutex_lock.part.1 () from /lib64/libkrb5.so.3
No symbol table info available.
#5  0x7fa7e8d7bda1 in k5_cc_mutex_lock () from /lib64/libkrb5.so.3
No symbol table info available.
#6  0x7fa7e8d851bf in krb5_mcc_store () from /lib64/libkrb5.so.3
No symbol table info available.
#7  0x7fa7e8d88070 in krb5_cc_store_cred () from /lib64/libkrb5.so.3
No symbol table info available.
#8  0x7fa7e8d98be3 in complete () from /lib64/libkrb5.so.3
No symbol table info available.
#9  0x7fa7e8d99ba1 in krb5_tkt_creds_step () from /lib64/libkrb5.so.3
No symbol table info available.
#10 0x7fa7e8d9a637 in krb5_tkt_creds_get () from /lib64/libkrb5.so.3
No symbol table info available.
#11 0x7fa7e8d9a75c in krb5_get_credentials () from /lib64/libkrb5.so.3
No symbol table info available.
#12 0x7fa7e0f6736a in krb5_gss_init_sec_context_ext () from 
/lib64/libgssapi_krb5.so.2

No symbol table info available.
#13 0x7fa7e0f67c97 in krb5_gss_init_sec_context () from 
/lib64/libgssapi_krb5.so.2

No symbol table info available.
#14 0x7fa7e0f516ab in gss_init_sec_context () from 
/lib64/libgssapi_krb5.so.2

No symbol table info available.
#15 0x7fa7e118ac44 in gssapi_client_mech_step () from 
/usr/lib64/sasl2/libgssapiv2.so

No symbol table info available.
#16 0x7fa7e72847f5 in sasl_client_step () from /lib64/libsasl2.so.3
No symbol table info available.
#17 0x7fa7e7284b76 in sasl_client_start () from /lib64/libsasl2.so.3
No symbol table info available.
#18 0x7fa7e8475e73 in ldap_int_sasl_bind () from 
/lib64/libldap_r-2.4.so.2

No symbol table info available.
#19 0x7fa7e8479492 in ldap_sasl_interactive_bind () from 
/lib64/libldap_r-2.4.so.2

No symbol table info available.
#20 0x7fa7e84796bd in ldap_sasl_interactive_bind_s () from 
/lib64/libldap_r-2.4.so.2


Directory server tries to open a replication connetion usig GSSAPI. I 
don't know which assertion fails in krb, but

- could you try with the replication agreement disabled ?
- Rob, you have been discussing renewals of keytabs, would we have to 
renew the ds.keytab ?



On 02/20/2016 07:57 AM, Timothy Geier wrote:

To update this thread..timeshifting to before the certs expired (Feb.
2) brings up the interesting issue of pki-tomcatd (sometimes) starting
while slapd crashes almost immediately after ipactl start is run..if it
doesn't crash right away, pki-tomcatd will start, but since slapd will
then go down right afterwards, the certs cannot be renewed.  After
returning to real time, slapd is just fine along with everything else
aside from pki-tomcatd.

A core dump is attached.


"This message and any attachments may contain confidential information. If you
have received this  message in error, any use or distribution is prohibited.
Please notify us by reply e-mail if you have mistakenly received this message,
and immediately and permanently delete it and any attachments. Thank you."




-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

[Rob Crittenden]

Timothy Geier wrote:
> 
>> On Feb 10, 2016, at 3:01 AM, Rob Crittenden > > wrote:
>>>
>>> [09/Feb/2016:12:55:41 -0600] conn=109598 fd=287 slot=287 SSL
>>> connection from master_ip to master_ip
>>> [09/Feb/2016:12:55:41 -0600] conn=109597 op=0 EXT
>>> oid="1.3.6.1.4.1.1466.20037" name="startTLS"
>>> [09/Feb/2016:12:55:41 -0600] conn=109597 op=0 RESULT err=0 tag=120
>>> nentries=0 etime=0
>>> [09/Feb/2016:12:55:41 -0600] conn=109598 Netscape Portable Runtime
>>> error -8181 (Peer's Certificate has expired.); unauthenticated client
>>> CN=CA Subsystem,O=XXX.NET ; issuer
>>> CN=Certificate Authority,O=XXX.NET 
>>> [09/Feb/2016:12:55:41 -0600] conn=109598 op=-1 fd=287 closed - Peer's
>>> Certificate has expired.
>>>
>>
>> Ok, right. The subsystem cert expired on Feb 1 so you'd have to back
>> at least that far in time to do the renewals.
>>
> 
>> There are a few entries in ou=People,o=ipaca that need to reflect the
>> current state of certificates as well.
> 
> Ah, right, just realized that that’s a base that can looked up
> separately in LDAP..is there anything in particular to look for in there?

Eventually you'll need to confirm that the IPA RA entry matches
correctly but if your CA isn't coming up at all it is putting the cart
before the horse.

> 
>>> 
>>>
>>> All of the host keytabs on all of the IPA servers are correct..are
>>> there any other keytabs to check?
>>
>> No, just /etc/krb5.keytab. I think you should focus on getting the CA
>> subsystem certs renewed and then we can look at the other things. So
>> I'd go back in time to Jan 30 or so and just restart certmonger.
> 
> After doing so, certmonger appears to run smoothly and goes from
> SUBMITTING to MONITORING but the expiration date on all of the certs
> stays the same. (It’s the same result if ipa-getcert resubmit is run
> against all of the request IDs..quite perplexing) 
> 
> If we do a total shutdown/rewind/restart, getcert list produces the
> following for these 3 certs during the time shift after the restart:
> 
> Request ID '20160209194022':
> status: CA_UNREACHABLE
> ca-error: Internal error
> stuck: no
> key pair storage:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
> cert-pki-ca',token='NSS Certificate DB',pin set
> certificate:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
> cert-pki-ca',token='NSS Certificate DB'
> CA: dogtag-ipa-ca-renew-agent
> issuer: CN=Certificate Authority,O=XXX.NET 
> subject: CN=CA Audit,O=XXX.NET 
> expires: 2016-02-01 19:46:48 UTC
> key usage: digitalSignature,nonRepudiation
> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert
> "auditSigningCert cert-pki-ca"
> track: yes
> auto-renew: yes
> Request ID '20160209194023':
> status: CA_UNREACHABLE
> ca-error: Internal error
> stuck: no
> key pair storage:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert 
> cert-pki-ca',token='NSS
> Certificate DB',pin set
> certificate:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert 
> cert-pki-ca',token='NSS
> Certificate DB'
> CA: dogtag-ipa-ca-renew-agent
> issuer: CN=Certificate Authority,O=XXX.NET 
> subject: CN=OCSP Subsystem,O=XXX.NET 
> expires: 2016-02-01 19:46:47 UTC
> key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
> eku: id-kp-OCSPSigning
> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert
> "ocspSigningCert cert-pki-ca"
> track: yes
> auto-renew: yes
> Request ID '20160209194024':
> status: CA_UNREACHABLE
> ca-error: Internal error
> stuck: no
> key pair storage:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
> cert-pki-ca',token='NSS Certificate DB',pin set
> certificate:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
> cert-pki-ca',token='NSS Certificate DB'
> CA: dogtag-ipa-ca-renew-agent
> issuer: CN=Certificate Authority,O=XXX.NET 
> subject: CN=CA Subsystem,O=XXX.NET 
> expires: 2016-02-01 19:46:47 UTC
> key usage:
> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> eku: id-kp-serverAuth,id-kp-clientAuth
> pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
> post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert
> "subsystemCert cert-pki-ca"
> track: yes
> auto-renew: yes
> 
> The 

Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

[Timothy Geier]

On Feb 10, 2016, at 3:01 AM, Rob Crittenden 
> wrote:

[09/Feb/2016:12:55:41 -0600] conn=109598 fd=287 slot=287 SSL connection from 
master_ip to master_ip
[09/Feb/2016:12:55:41 -0600] conn=109597 op=0 EXT oid="1.3.6.1.4.1.1466.20037" 
name="startTLS"
[09/Feb/2016:12:55:41 -0600] conn=109597 op=0 RESULT err=0 tag=120 nentries=0 
etime=0
[09/Feb/2016:12:55:41 -0600] conn=109598 Netscape Portable Runtime error -8181 
(Peer's Certificate has expired.); unauthenticated client CN=CA 
Subsystem,O=XXX.NET; issuer CN=Certificate 
Authority,O=XXX.NET
[09/Feb/2016:12:55:41 -0600] conn=109598 op=-1 fd=287 closed - Peer's 
Certificate has expired.


Ok, right. The subsystem cert expired on Feb 1 so you'd have to back at least 
that far in time to do the renewals.


There are a few entries in ou=People,o=ipaca that need to reflect the current 
state of certificates as well.

Ah, right, just realized that that’s a base that can looked up separately in 
LDAP..is there anything in particular to look for in there?




All of the host keytabs on all of the IPA servers are correct..are there any 
other keytabs to check?

No, just /etc/krb5.keytab. I think you should focus on getting the CA subsystem 
certs renewed and then we can look at the other things. So I'd go back in time 
to Jan 30 or so and just restart certmonger.

After doing so, certmonger appears to run smoothly and goes from SUBMITTING to 
MONITORING but the expiration date on all of the certs stays the same. (It’s 
the same result if ipa-getcert resubmit is run against all of the request 
IDs..quite perplexing)

If we do a total shutdown/rewind/restart, getcert list produces the following 
for these 3 certs during the time shift after the restart:

Request ID '20160209194022':
status: CA_UNREACHABLE
ca-error: Internal error
stuck: no
key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=XXX.NET
subject: CN=CA Audit,O=XXX.NET
expires: 2016-02-01 19:46:48 UTC
key usage: digitalSignature,nonRepudiation
pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert 
"auditSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160209194023':
status: CA_UNREACHABLE
ca-error: Internal error
stuck: no
key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert 
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=XXX.NET
subject: CN=OCSP Subsystem,O=XXX.NET
expires: 2016-02-01 19:46:47 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
eku: id-kp-OCSPSigning
pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert 
"ocspSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160209194024':
status: CA_UNREACHABLE
ca-error: Internal error
stuck: no
key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert 
cert-pki-ca',token='NSS Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert 
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=XXX.NET
subject: CN=CA Subsystem,O=XXX.NET
expires: 2016-02-01 19:46:47 UTC
key usage: 
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert 
"subsystemCert cert-pki-ca"
track: yes
auto-renew: yes

The most puzzling thing (in my opinion) about this issue is that timeshifting 
doesn’t seem to make any difference at all; pki-tomcatd still doesn’t start 
cleanly (the log entries are very similar) even though in theory those certs 
are no longer expired at that point..it seems as if something else is also the 
issue.

Your ongoing assistance with this matter is much appreciated.



rob





"This message and any 

Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

[Rob Crittenden]

Timothy Geier wrote:



On Feb 9, 2016, at 2:58 AM, Rob Crittenden  wrote:

Timothy Geier wrote:



The debug log has a lot of instances of:

Could not connect to LDAP server host xxx. port 636 Error
netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1)
Internal Database Error encountered: Could not connect to LDAP server
host xxx. port 636 Error netscape.ldap.LDAPException: IO Error
creating JSS SSL Socket (-1)


The 389-ds access log may show connection attempts and you might be able to 
correlate from there.



[09/Feb/2016:12:55:41 -0600] conn=109598 fd=287 slot=287 SSL connection from 
master_ip to master_ip
[09/Feb/2016:12:55:41 -0600] conn=109597 op=0 EXT oid="1.3.6.1.4.1.1466.20037" 
name="startTLS"
[09/Feb/2016:12:55:41 -0600] conn=109597 op=0 RESULT err=0 tag=120 nentries=0 
etime=0
[09/Feb/2016:12:55:41 -0600] conn=109598 Netscape Portable Runtime error -8181 
(Peer's Certificate has expired.); unauthenticated client CN=CA 
Subsystem,O=XXX.NET; issuer CN=Certificate Authority,O=XXX.NET
[09/Feb/2016:12:55:41 -0600] conn=109598 op=-1 fd=287 closed - Peer's 
Certificate has expired.



Ok, right. The subsystem cert expired on Feb 1 so you'd have to back at 
least that far in time to do the renewals.







Some data is also stored in CS.cfg so you might ensure that the certificates 
stored there are correct as well.



CS.cfg is correct/restored from backup.


There are a few entries in ou=People,o=ipaca that need to reflect the current 
state of certificates as well.



While looking in cn=config for any o=ipaca entries, the following popped up..it 
looks like most of the old replicas and the old master hostnames weren’t 
properly cleaned up and there’s still RUVs for them..nsDS5ReplicaHost and Port 
also seem to be improperly set:


A problem for a future day I think.


ou=People doesn’t seem to exist in either the main tree or cn=config.


The base is ou=People,o=ipaca


Ok, you'll need to look at the host keytabs because something seems wrong with 
them. certmonger can't get a ticket using them. This could be because IPA isn't 
fully running but it is worth a look.


All of the host keytabs on all of the IPA servers are correct..are there any 
other keytabs to check?


No, just /etc/krb5.keytab. I think you should focus on getting the CA 
subsystem certs renewed and then we can look at the other things. So I'd 
go back in time to Jan 30 or so and just restart certmonger.


rob






Is this the correct procedure to follow when time shifted?

- Stop IPA
- Change the system clock (and the hardware clock) to a point before the
expiration
- Start IPA
- Run getcert resubmit on the appropriate request IDs
- Stop IPA
- Return to real time
- Start IPA

We haven’t tried it this week yet but all attempts at it last week
failed without any indication as to why the certs weren’t renewing; are
there any other logs to check/other steps in the procedure?


Yes, that is correct.



Thanks for the clarification.  The IPA cluster itself is still doing ok after 
about a week of being in this state but something else we’re wondering is if 
we’ll be able to add replica servers while this is going on..it’s not critical 
to do so, but it is going to come up fairly soon.


rob




"This message and any attachments may contain confidential information. If you
have received this  message in error, any use or distribution is prohibited.
Please notify us by reply e-mail if you have mistakenly received this message,
and immediately and permanently delete it and any attachments. Thank you."



--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

[Timothy Geier]

> On Feb 9, 2016, at 2:58 AM, Rob Crittenden  wrote:
> 
> Timothy Geier wrote:
>> 
>> 
>> The debug log has a lot of instances of:
>> 
>> Could not connect to LDAP server host xxx. port 636 Error
>> netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1)
>> Internal Database Error encountered: Could not connect to LDAP server
>> host xxx. port 636 Error netscape.ldap.LDAPException: IO Error
>> creating JSS SSL Socket (-1)
> 
> The 389-ds access log may show connection attempts and you might be able to 
> correlate from there.
> 

[09/Feb/2016:12:55:41 -0600] conn=109598 fd=287 slot=287 SSL connection from 
master_ip to master_ip
[09/Feb/2016:12:55:41 -0600] conn=109597 op=0 EXT oid="1.3.6.1.4.1.1466.20037" 
name="startTLS"
[09/Feb/2016:12:55:41 -0600] conn=109597 op=0 RESULT err=0 tag=120 nentries=0 
etime=0
[09/Feb/2016:12:55:41 -0600] conn=109598 Netscape Portable Runtime error -8181 
(Peer's Certificate has expired.); unauthenticated client CN=CA 
Subsystem,O=XXX.NET; issuer CN=Certificate Authority,O=XXX.NET
[09/Feb/2016:12:55:41 -0600] conn=109598 op=-1 fd=287 closed - Peer's 
Certificate has expired.



> 
> Some data is also stored in CS.cfg so you might ensure that the certificates 
> stored there are correct as well.
> 

CS.cfg is correct/restored from backup.

> There are a few entries in ou=People,o=ipaca that need to reflect the current 
> state of certificates as well.
> 

While looking in cn=config for any o=ipaca entries, the following popped up..it 
looks like most of the old replicas and the old master hostnames weren’t 
properly cleaned up and there’s still RUVs for them..nsDS5ReplicaHost and Port 
also seem to be improperly set:

cn=cloneAgreement1-current-master.X.net-pki-tomcat,cn=replica,cn=o\\3Dipaca,cn=mapping
 tree,cn=config
cn: cloneAgreement1-current-master.X.net-pki-tomcat
description: cloneAgreement1-current-master.X.net-pki-tomcat
nsDS5ReplicaBindDN: cn=Replication Manager 
masterAgreement1-current-master.X.net-pki-tomcat,ou=csusers,cn=config
nsDS5ReplicaBindMethod: Simple
nsDS5ReplicaHost: old-master.X.net
nsDS5ReplicaPort: 7389
nsDS5ReplicaRoot: o=ipaca
nsDS5ReplicaTransportInfo: TLS
nsds50ruv: {replicageneration} 5304e0780060
nsds50ruv: {replica 96 ldap://old-master.X.net:7389} 
5304e0840060 561f3e8f00050060
nsds50ruv: {replica 81 ldap://current-master.X.net:389} 
561f361b0051 561f369c0051
nsds50ruv: {replica 86 ldap://old-replica3.X.net:7389} 
53f7b70d0056 5616ad6400090056
nsds50ruv: {replica 91 ldap://old-replica2.X.net:7389} 
53a841e5005b 5616ad5f0005005b
nsds50ruv: {replica 97 ldap://old-replica2.X.net:7389} 
5304e0800061 539b4f9600010061
nsruvReplicaLastModified: {replica 96 ldap://old-master.X.net:7389} 

nsruvReplicaLastModified: {replica 81 ldap://current-master.X.net:389} 

nsruvReplicaLastModified: {replica 86 ldap://old-replica3.X.net:7389} 

nsruvReplicaLastModified: {replica 91 ldap://old-replica2.X.net:7389} 

nsruvReplicaLastModified: {replica 97 ldap://old-replica2.X.net:7389} 

objectClass: top
objectClass: nsds5replicationagreement
nsds5replicareapactive: 0
nsds5replicaLastUpdateStart: 1970010100Z
nsds5replicaLastUpdateEnd: 1970010100Z
nsds5replicaChangesSentSinceStartup:
nsds5replicaLastUpdateStatus: -1 Unable to acquire replicaLDAP error: Can't 
contact LDAP server
nsds5replicaUpdateInProgress: FALSE
nsds5replicaLastInitStart: 1970010100Z
nsds5replicaLastInitEnd: 1970010100Z

cn=replica,cn=o\\3Dipaca,cn=mapping tree,cn=config
cn: replica
nsDS5Flags: 1
nsDS5ReplicaBindDN: cn=Replication Manager 
cloneAgreement1-current-master.X.net-pki-tomcat,ou=csusers,cn=config
nsDS5ReplicaId: 81
nsDS5ReplicaName: 56db714e-72e411e5-87dbe691-e12b51f2
nsDS5ReplicaRoot: o=ipaca
nsDS5ReplicaType: 3
nsState:: UQDkWbdWwCYBAA==
objectClass: top
objectClass: nsDS5Replica
objectClass: extensibleobject
nsds5ReplicaChangeCount: 11463
nsds5replicareapactive: 0

ou=People doesn’t seem to exist in either the main tree or cn=config.

>> 
>>> You mentioned privately that you renamed the IPA host. This is
>>> probably what broke half of the renewals. The hosts and keytabs and
>>> many entries in IPA have the hostname baked in so you can't simply
>>> rename the host.
>>> 
>> 
>> Technically, the host wasn’t renamed; a new CentOS 7 host was added to
>> the existing IPA cluster that had 4 hosts (1 master CA) at CentOS 6
>> (using the documentation at
>> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/upgrading.html),
>> it was promoted to the master CA, all of the C6 hosts were
>> decommissioned/removed from replication, and then a new set of C7 hosts
>> were created 

Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

[Rob Crittenden]

Timothy Geier wrote:



The debug log has a lot of instances of:

Could not connect to LDAP server host xxx. port 636 Error
netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1)
Internal Database Error encountered: Could not connect to LDAP server
host xxx. port 636 Error netscape.ldap.LDAPException: IO Error
creating JSS SSL Socket (-1)


The 389-ds access log may show connection attempts and you might be able 
to correlate from there.



but nothing else of note other than those errors.

We’ve also noticed lots of 500 errors in
/var/log/pki/pki-tomcat/localhost_access.log
[08/Feb/2016:10:34:29 -0600] "GET
/ca/ee/ca/profileSubmit?profileId=caServerCert_num=5=true=true
HTTP/1.1" 500 2134
[08/Feb/2016:10:34:32 -0600] "GET
/ca/ee/ca/profileSubmit?profileId=caServerCert_num=2=true=true
HTTP/1.1" 500 2134
[08/Feb/2016:10:34:50 -0600] "GET
/ca/ee/ca/profileSubmit?profileId=caServerCert_num=4=true=true
HTTP/1.1" 500 2134

which looks like certmonger is continuously trying to renew the 3 certs.

These dates and times from selftests.log are not accurate and are from
an earlier attempt to renew the certs while time shifted:

0.localhost-startStop-1 - [15/Jan/2016:17:39:34 CST] [20] [1]
SelfTestSubsystem: Initializing self test plugins:
0.localhost-startStop-1 - [15/Jan/2016:17:39:34 CST] [20] [1]
SelfTestSubsystem:  loading all self test plugin logger parameters
0.localhost-startStop-1 - [15/Jan/2016:17:39:34 CST] [20] [1]
SelfTestSubsystem:  loading all self test plugin instances
0.localhost-startStop-1 - [15/Jan/2016:17:39:34 CST] [20] [1]
SelfTestSubsystem:  loading all self test plugin instance parameters
0.localhost-startStop-1 - [15/Jan/2016:17:39:34 CST] [20] [1]
SelfTestSubsystem:  loading self test plugins in on-demand order
0.localhost-startStop-1 - [15/Jan/2016:17:39:34 CST] [20] [1]
SelfTestSubsystem:  loading self test plugins in startup order
0.localhost-startStop-1 - [15/Jan/2016:17:39:34 CST] [20] [1]
SelfTestSubsystem: Self test plugins have been successfully loaded!
0.localhost-startStop-1 - [15/Jan/2016:17:39:38 CST] [20] [1]
SelfTestSubsystem: Running self test plugins specified to be executed at
startup:
0.localhost-startStop-1 - [15/Jan/2016:17:39:38 CST] [20] [1]
CAPresence:  CA is present
0.localhost-startStop-1 - [15/Jan/2016:17:39:38 CST] [20] [1]
SystemCertsVerification: system certs verification success
0.localhost-startStop-1 - [15/Jan/2016:17:39:38 CST] [20] [1]
SelfTestSubsystem: All CRITICAL self test plugins ran SUCCESSFULLY at
startup!

There’s nothing in that log with any February dates, so when we try to
start pki-tomcatd in real time, it's likely not even getting this far.


Some data is also stored in CS.cfg so you might ensure that the 
certificates stored there are correct as well.


There are a few entries in ou=People,o=ipaca that need to reflect the 
current state of certificates as well.





You mentioned privately that you renamed the IPA host. This is
probably what broke half of the renewals. The hosts and keytabs and
many entries in IPA have the hostname baked in so you can't simply
rename the host.



Technically, the host wasn’t renamed; a new CentOS 7 host was added to
the existing IPA cluster that had 4 hosts (1 master CA) at CentOS 6
(using the documentation at
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/upgrading.html),
it was promoted to the master CA, all of the C6 hosts were
decommissioned/removed from replication, and then a new set of C7 hosts
were created and added as replicas.


Ok, you'll need to look at the host keytabs because something seems 
wrong with them. certmonger can't get a ticket using them. This could be 
because IPA isn't fully running but it is worth a look.



Is this the correct procedure to follow when time shifted?

- Stop IPA
- Change the system clock (and the hardware clock) to a point before the
expiration
- Start IPA
- Run getcert resubmit on the appropriate request IDs
- Stop IPA
- Return to real time
- Start IPA

We haven’t tried it this week yet but all attempts at it last week
failed without any indication as to why the certs weren’t renewing; are
there any other logs to check/other steps in the procedure?


Yes, that is correct.

rob

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

[Rob Crittenden]

Timothy Geier wrote:

Greetings all,

For the record,this is a CentOS 7.2 box with all current patches. 
(ipa-server-4.2.0-15.el7.centos.3.x86_64, etc.)

The situation is that pki-tomcatd on the lone CA server in our IPA cluster 
refuses to start cleanly.  The issues started earlier this week after the certs
subsystemCert, ocspSigningCert, and auditSigningCert all simultaneously expired 
without warning; apparently, certmonger failed to renew them automatically.  We
attempted timeshifting and following instructions for what appeared to be 
similar issues, but nothing at all has worked.

Today, we attempted removing the certificates in question (of course, the files in 
/etc/pki/pki-tomcat/alias were backed up beforehand) and using certutil to issue new  
certificates.   This process worked but pki-tomcatd is still refusing to start.  We can 
get IPA to run on this server by manually starting pki-tomcatd, running ipactl start, and 
then ctrl-c’ing it when it gets to "Starting pki-tomcatd" but this is not a 
tenable long-term solution.

Relevant log entries/information:

/var/log/pki/pki-tomcat/ca/debug:
Could not connect to LDAP server host ipa01.X.net port 636 Error 
netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1)
Internal Database Error encountered: Could not connect to LDAP server host 
ipa01.X.net port 636 Error netscape.ldap.LDAPException: IO Error 
creating JSS SSL Socket (-1)
Internal Database Error encountered: Could not connect to LDAP server host 
ipa01.X.net port 636 Error netscape.ldap.LDAPException: Authentication 
failed (49)

/var/log/pki/pki-tomcat/localhost.2016-02-04.log:
org.apache.catalina.core.StandardContext loadOnStartup
SEVERE: Servlet /ca threw load() exception
java.lang.NullPointerException

# getcert list:

Number of certificates and requests being tracked: 8.
Request ID '20151015022737':
status: MONITORING
ca-error: Error setting up ccache for "host" service on client using 
default keytab: Generic error (see e-text).
stuck: no
key pair storage: 
type=NSSDB,location='/etc/dirsrv/slapd-X-NET',nickname='Server-Cert',token='NSS
 Certificate DB',pinfile='/etc/dirsrv/slapd-X-NET/pwdfile.txt'
expires: 2017-10-15 02:09:06 UTC
track: yes
auto-renew: yes
Request ID '20151015022949':
status: MONITORING
ca-error: Error setting up ccache for "host" service on client using 
default keytab: Generic error (see e-text).
stuck: no
key pair storage: 
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS 
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: 
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS 
Certificate DB'
expires: 2017-10-15 02:09:10 UTC
track: yes
auto-renew: yes
Request ID '20160127202548':
status: MONITORING
stuck: no
key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert 
cert-pki-ca',token='NSS Certificate DB'
expires: 2034-02-11 19:46:43 UTC
track: yes
auto-renew: yes
Request ID '20160127202549':
status: MONITORING
stuck: no
key pair storage: 
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS 
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: 
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS 
Certificate DB'
expires: 2017-12-25 04:27:49 UTC
key usage: 
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
track: yes
auto-renew: yes
Request ID '20160127202550':
status: MONITORING
ca-error: Server at 
"http://ipa01.X.net:8080/ca/ee/ca/profileSubmit; replied: Profile 
caServerCert Not Found
stuck: no
key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert 
cert-pki-ca',token='NSS Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert 
cert-pki-ca',token='NSS Certificate DB'
expires: 2017-10-04 02:28:53 UTC
track: yes
auto-renew: yes
Request ID '20160204165453':
status: MONITORING
stuck: no
key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
cert-pki-ca',token='NSS Certificate DB'
expires: 2016-05-04 16:40:23 UTC
track: yes
auto-renew: yes
Request ID '20160204170246':
status: MONITORING
stuck: no
key pair storage: 

Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

[Timothy Geier]

On Feb 8, 2016, at 4:28 AM, Rob Crittenden 
> wrote:

Timothy Geier wrote:
Greetings all,

For the record,this is a CentOS 7.2 box with all current patches. 
(ipa-server-4.2.0-15.el7.centos.3.x86_64, etc.)

The situation is that pki-tomcatd on the lone CA server in our IPA cluster 
refuses to start cleanly.  The issues started earlier this week after the certs
subsystemCert, ocspSigningCert, and auditSigningCert all simultaneously expired 
without warning; apparently, certmonger failed to renew them automatically.  We
attempted timeshifting and following instructions for what appeared to be 
similar issues, but nothing at all has worked.

Today, we attempted removing the certificates in question (of course, the files 
in /etc/pki/pki-tomcat/alias were backed up beforehand) and using certutil to 
issue new  certificates.   This process worked but pki-tomcatd is still 
refusing to start.  We can get IPA to run on this server by manually starting 
pki-tomcatd, running ipactl start, and then ctrl-c’ing it when it gets to 
"Starting pki-tomcatd" but this is not a tenable long-term solution.

Relevant log entries/information:

/var/log/pki/pki-tomcat/ca/debug:
Could not connect to LDAP server host 
ipa01.X.net port 636 Error 
netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1)
Internal Database Error encountered: Could not connect to LDAP server host 
ipa01.X.net port 636 Error 
netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1)
Internal Database Error encountered: Could not connect to LDAP server host 
ipa01.X.net port 636 Error 
netscape.ldap.LDAPException: Authentication failed (49)

/var/log/pki/pki-tomcat/localhost.2016-02-04.log:
org.apache.catalina.core.StandardContext loadOnStartup
SEVERE: Servlet /ca threw load() exception
java.lang.NullPointerException

# getcert list:

Number of certificates and requests being tracked: 8.
Request ID '20151015022737':
status: MONITORING
ca-error: Error setting up ccache for "host" service on client using default 
keytab: Generic error (see e-text).
stuck: no
key pair storage: 
type=NSSDB,location='/etc/dirsrv/slapd-X-NET',nickname='Server-Cert',token='NSS
 Certificate DB',pinfile='/etc/dirsrv/slapd-X-NET/pwdfile.txt'
expires: 2017-10-15 02:09:06 UTC
track: yes
auto-renew: yes
Request ID '20151015022949':
status: MONITORING
ca-error: Error setting up ccache for "host" service on client using default 
keytab: Generic error (see e-text).
stuck: no
key pair storage: 
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS 
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: 
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS 
Certificate DB'
expires: 2017-10-15 02:09:10 UTC
track: yes
auto-renew: yes
Request ID '20160127202548':
status: MONITORING
stuck: no
key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert 
cert-pki-ca',token='NSS Certificate DB'
expires: 2034-02-11 19:46:43 UTC
track: yes
auto-renew: yes
Request ID '20160127202549':
status: MONITORING
stuck: no
key pair storage: 
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS 
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: 
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS 
Certificate DB'
expires: 2017-12-25 04:27:49 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
track: yes
auto-renew: yes
Request ID '20160127202550':
status: MONITORING
ca-error: Server at "http://ipa01.X.net:8080/ca/ee/ca/profileSubmit; 
replied: Profile caServerCert Not Found
stuck: no
key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert 
cert-pki-ca',token='NSS Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert 
cert-pki-ca',token='NSS Certificate DB'
expires: 2017-10-04 02:28:53 UTC
track: yes
auto-renew: yes
Request ID '20160204165453':
status: MONITORING
stuck: no
key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
cert-pki-ca',token='NSS Certificate DB'
expires: 2016-05-04 16:40:23 UTC
track: yes
auto-renew: yes
Request ID '20160204170246':
status: MONITORING
stuck: no
key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert 
cert-pki-ca',token='NSS Certificate DB'
expires: 

Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

[Rob Crittenden]

Timothy Geier wrote:

Greetings all,

For the record,this is a CentOS 7.2 box with all current patches. 
(ipa-server-4.2.0-15.el7.centos.3.x86_64, etc.)

The situation is that pki-tomcatd on the lone CA server in our IPA cluster 
refuses to start cleanly.  The issues started earlier this week after the certs
subsystemCert, ocspSigningCert, and auditSigningCert all simultaneously expired 
without warning; apparently, certmonger failed to renew them automatically.  We
attempted timeshifting and following instructions for what appeared to be 
similar issues, but nothing at all has worked.

Today, we attempted removing the certificates in question (of course, the files in 
/etc/pki/pki-tomcat/alias were backed up beforehand) and using certutil to issue new  
certificates.   This process worked but pki-tomcatd is still refusing to start.  We can 
get IPA to run on this server by manually starting pki-tomcatd, running ipactl start, and 
then ctrl-c’ing it when it gets to "Starting pki-tomcatd" but this is not a 
tenable long-term solution.

Relevant log entries/information:

/var/log/pki/pki-tomcat/ca/debug:
Could not connect to LDAP server host ipa01.X.net port 636 Error 
netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1)
Internal Database Error encountered: Could not connect to LDAP server host 
ipa01.X.net port 636 Error netscape.ldap.LDAPException: IO Error 
creating JSS SSL Socket (-1)
Internal Database Error encountered: Could not connect to LDAP server host 
ipa01.X.net port 636 Error netscape.ldap.LDAPException: Authentication 
failed (49)

/var/log/pki/pki-tomcat/localhost.2016-02-04.log:
org.apache.catalina.core.StandardContext loadOnStartup
SEVERE: Servlet /ca threw load() exception
java.lang.NullPointerException

# getcert list:

Number of certificates and requests being tracked: 8.
Request ID '20151015022737':
status: MONITORING
ca-error: Error setting up ccache for "host" service on client using 
default keytab: Generic error (see e-text).
stuck: no
key pair storage: 
type=NSSDB,location='/etc/dirsrv/slapd-X-NET',nickname='Server-Cert',token='NSS
 Certificate DB',pinfile='/etc/dirsrv/slapd-X-NET/pwdfile.txt'
expires: 2017-10-15 02:09:06 UTC
track: yes
auto-renew: yes
Request ID '20151015022949':
status: MONITORING
ca-error: Error setting up ccache for "host" service on client using 
default keytab: Generic error (see e-text).
stuck: no
key pair storage: 
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS 
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: 
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS 
Certificate DB'
expires: 2017-10-15 02:09:10 UTC
track: yes
auto-renew: yes
Request ID '20160127202548':
status: MONITORING
stuck: no
key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert 
cert-pki-ca',token='NSS Certificate DB'
expires: 2034-02-11 19:46:43 UTC
track: yes
auto-renew: yes
Request ID '20160127202549':
status: MONITORING
stuck: no
key pair storage: 
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS 
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: 
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS 
Certificate DB'
expires: 2017-12-25 04:27:49 UTC
key usage: 
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
track: yes
auto-renew: yes
Request ID '20160127202550':
status: MONITORING
ca-error: Server at 
"http://ipa01.X.net:8080/ca/ee/ca/profileSubmit; replied: Profile 
caServerCert Not Found
stuck: no
key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert 
cert-pki-ca',token='NSS Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert 
cert-pki-ca',token='NSS Certificate DB'
expires: 2017-10-04 02:28:53 UTC
track: yes
auto-renew: yes
Request ID '20160204165453':
status: MONITORING
stuck: no
key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
cert-pki-ca',token='NSS Certificate DB'
expires: 2016-05-04 16:40:23 UTC
track: yes
auto-renew: yes
Request ID '20160204170246':
status: MONITORING
stuck: no
key pair storage: