[Freeipa-users] IPA to IPA migration

2017-01-04 Thread Timothy Geier 
This is something I’ve looked at lately and a manual proof of concept I just 
did (using ideas from 
https://www.freeipa.org/page/Howto/Migration#Migrating_from_other_FreeIPA_to_FreeIPA)
 makes it seem theoretically possible (though it looks like, barring the 
migration of the kerberos master key, all enrolled hosts would need to use 
ipa-getkeytab to get a replacement keytab from the new server and copy it to 
/etc/krb5.keytab so that sssd will work properly..the alternative is 
re-enrollment.  All other keytabs in use by other applications would have to be 
similarly replaced).

Is https://fedorahosted.org/freeipa/ticket/3656 something that’s coming sooner 
or later to a future version of FreeIPA?  Has anyone done a manual migration on 
a moderate-to-large setup?
-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Replication broken

2016-09-27 Thread Timothy Geier 
On Tue, 2016-09-27 at 12:47 +0200, thierry bordaz wrote:
> Hi Timothy,
> 
> The changenumber counter is protected by a lock and we should not see
> duplicate value.. except if there is a bug :-(
> 
> Retrieving the time when changenumber=112697,cn=changelog was created
> and the time when you saw the error, can you see any error in
> operations (access log) or in the error log ?
> 
> Or did you disabled/enable retorCL between those two times ?
> 
> regards
> thiery

Unfortunately, the issue appears to be a certain username that starts
with a '1'..in both cases, trying to delete this user caused (and is
causing) the exact same issue.  Are there any known bugs relating to
this?

> 
> 
> 
> On 09/27/2016 12:37 AM, Timothy Geier wrote:
> 
> > 
> > > On Sep 26, 2016, at 4:07 PM, Timothy Geier <tge...@accertify.com>
> > > wrote:
> > > 
> > > > 
> > > > On Sep 26, 2016, at 2:17 PM, Timothy Geier
> > > > <tge...@accertify.com> wrote:
> > > > 
> > > > This issue started when trying to remove a user; ipa user-del
> > > > showed “operation failed” and the user was not removed.  The
> > > > same ipa user-del command was performed on a replica and
> > > > completed successfully, but it was then immediately apparent
> > > > that this change did not replicate anywhere else.  All of the
> > > > replicas then were re-initalized using "ipa-replica-manage
> > > > re-initialize” and now the LDAP trees/users are consistent
> > > > though no further changes have been made.
> > > > 
> > > > The slapd error logs are showing repeated instances of
> > > > 
> > > > DSRetroclPlugin - replog: an error occured while adding change
> > > > number 112697, dn = changenumber=112697,cn=changelog: Already
> > > > exists.
> > > > retrocl-plugin - retrocl_postob: operation failure [68]
> > > > 
> > > > Package versions are
> > > > ipa-server-4.2.0-15.0.1.el7.centos.6.1.x86_64
> > > > and 
> > > > 389-ds-base-1.3.4.0-29.el7_2.x86_64
> > > > 
> > > > ipa-replica-manage list-ruv
> > > > ipa: WARNING: session memcached servers not running
> > > > unable to decode: {replica 11} 56044ef5000b
> > > > 56044ef5000b
> > > > unable to decode: {replica 7} 561f17ba00080007
> > > > 561f17ba00080007
> > > > unable to decode: {replica 5} 561f17bc00030005
> > > > 561f17bc00030005
> > > > unable to decode: {replica 9} 561f17ba000a0009
> > > > 561f17ba000a0009
> > > > unable to decode: {replica 4} 561f17ba00030004
> > > > 561f17ba00030004 
> > > > (These are likely leftovers from the previous incarnation of
> > > > these servers on a RHEL6-like setup)
> > > > ipa07:389: 16
> > > > ipa02:389: 13
> > > > ipa03:389: 14
> > > > ipa01:389: 12
> > > > ipa04:389: 15
> > > > ipa05:389: 17
> > > > 
> > > > Thanks much,
> > > 
> > > After not taking any action, this error has stopped but has been
> > > replaced with
> > > 
> > > [26/Sep/2016:15:54:54 -0500] NSMMReplicationPlugin -
> > > agmt="cn=meToipa03" (ipa03:389): Missing data encountered
> > > [26/Sep/2016:15:54:54 -0500] NSMMReplicationPlugin -
> > > agmt="cn=meToipa03" (ipa03:389): Incremental update failed and
> > > requires administrator action
> > > 
> > > for all of the replicas and things are slightly out of sync
> > > everywhere.  
> > > 
> > > Is the best course of action here to declare one a new master and
> > > do a ipa-replica-manage re-initialize to all of the others from
> > > that one?
> > > 
> > > 
> > > 
> > 
> > 
> > After doing some testing, that’s exactly what we did and replication
> > is now working again.  It is odd that the DSRetroclPlugin errors
> > stopped on their own (after approximately 3 hours); the only action
> > taken there was looking at the cn=changelog base using ldapvi to see
> > what number it was on but that has to be a sheer coincidence;
> > absolutely no changes were made. 
> > 
> > 
> > We’re also still unsure what caused this; our best theory at the
> > moment is a race condition where everything that could have gone
> > wrong at that exact moment did..is there any validity to this?
> > 
> >

Re: [Freeipa-users] Replication broken

2016-09-26 Thread Timothy Geier 

On Sep 26, 2016, at 4:07 PM, Timothy Geier 
<tge...@accertify.com<mailto:tge...@accertify.com>> wrote:


On Sep 26, 2016, at 2:17 PM, Timothy Geier 
<tge...@accertify.com<mailto:tge...@accertify.com>> wrote:

This issue started when trying to remove a user; ipa user-del showed “operation 
failed” and the user was not removed.  The same ipa user-del command was 
performed on a replica and completed successfully, but it was then immediately 
apparent that this change did not replicate anywhere else.  All of the replicas 
then were re-initalized using "ipa-replica-manage re-initialize” and now the 
LDAP trees/users are consistent though no further changes have been made.

The slapd error logs are showing repeated instances of

DSRetroclPlugin - replog: an error occured while adding change number 112697, 
dn = changenumber=112697,cn=changelog: Already exists.
retrocl-plugin - retrocl_postob: operation failure [68]

Package versions are
ipa-server-4.2.0-15.0.1.el7.centos.6.1.x86_64
and
389-ds-base-1.3.4.0-29.el7_2.x86_64

ipa-replica-manage list-ruv
ipa: WARNING: session memcached servers not running
unable to decode: {replica 11} 56044ef5000b 56044ef5000b
unable to decode: {replica 7} 561f17ba00080007 561f17ba00080007
unable to decode: {replica 5} 561f17bc00030005 561f17bc00030005
unable to decode: {replica 9} 561f17ba000a0009 561f17ba000a0009
unable to decode: {replica 4} 561f17ba00030004 561f17ba00030004
(These are likely leftovers from the previous incarnation of these servers on a 
RHEL6-like setup)
ipa07:389: 16
ipa02:389: 13
ipa03:389: 14
ipa01:389: 12
ipa04:389: 15
ipa05:389: 17

Thanks much,

After not taking any action, this error has stopped but has been replaced with

[26/Sep/2016:15:54:54 -0500] NSMMReplicationPlugin - agmt="cn=meToipa03" 
(ipa03:389): Missing data encountered
[26/Sep/2016:15:54:54 -0500] NSMMReplicationPlugin - agmt="cn=meToipa03" 
(ipa03:389): Incremental update failed and requires administrator action

for all of the replicas and things are slightly out of sync everywhere.

Is the best course of action here to declare one a new master and do a 
ipa-replica-manage re-initialize to all of the others from that one?



After doing some testing, that’s exactly what we did and replication is now 
working again.  It is odd that the DSRetroclPlugin errors stopped on their own 
(after approximately 3 hours); the only action taken there was looking at the 
cn=changelog base using ldapvi to see what number it was on but that has to be 
a sheer coincidence; absolutely no changes were made.

We’re also still unsure what caused this; our best theory at the moment is a 
race condition where everything that could have gone wrong at that exact moment 
did..is there any validity to this?

Thanks,



"This message and any attachments may contain confidential information. If you
have received this  message in error, any use or distribution is prohibited. 
Please notify us by reply e-mail if you have mistakenly received this message,
and immediately and permanently delete it and any attachments. Thank you."-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

[Freeipa-users] Replication broken

2016-09-26 Thread Timothy Geier 
This issue started when trying to remove a user; ipa user-del showed “operation 
failed” and the user was not removed.  The same ipa user-del command was 
performed on a replica and completed successfully, but it was then immediately 
apparent that this change did not replicate anywhere else.  All of the replicas 
then were re-initalized using "ipa-replica-manage re-initialize” and now the 
LDAP trees/users are consistent though no further changes have been made.

The slapd error logs are showing repeated instances of

DSRetroclPlugin - replog: an error occured while adding change number 112697, 
dn = changenumber=112697,cn=changelog: Already exists.
retrocl-plugin - retrocl_postob: operation failure [68]

Package versions are
ipa-server-4.2.0-15.0.1.el7.centos.6.1.x86_64
and 
389-ds-base-1.3.4.0-29.el7_2.x86_64

ipa-replica-manage list-ruv
ipa: WARNING: session memcached servers not running
unable to decode: {replica 11} 56044ef5000b 56044ef5000b
unable to decode: {replica 7} 561f17ba00080007 561f17ba00080007
unable to decode: {replica 5} 561f17bc00030005 561f17bc00030005
unable to decode: {replica 9} 561f17ba000a0009 561f17ba000a0009
unable to decode: {replica 4} 561f17ba00030004 561f17ba00030004 
(These are likely leftovers from the previous incarnation of these servers on a 
RHEL6-like setup)
ipa07:389: 16
ipa02:389: 13
ipa03:389: 14
ipa01:389: 12
ipa04:389: 15
ipa05:389: 17

Thanks much,


"This message and any attachments may contain confidential information. If you
have received this  message in error, any use or distribution is prohibited. 
Please notify us by reply e-mail if you have mistakenly received this message,
and immediately and permanently delete it and any attachments. Thank you."

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

2016-03-29 Thread Timothy Geier 

> On Mar 29, 2016, at 2:00 AM, Thorsten Scherf <tsch...@redhat.com> wrote:
> 
> On [Mon, 28.03.2016 18:18], Timothy Geier wrote:
>> 
>>> On Mar 28, 2016, at 12:53 PM, Thorsten Scherf <tsch...@redhat.com> wrote:
>>> 
>>> On [Sat, 26.03.2016 03:26], Timothy Geier wrote:
>>>> To follow up on this issue, we haven’t been able to get any further since
>>>> last month due to the missing caServerCert profile..the configuration
>>>> files /usr/share/pki/ca/profiles/ca/caServerCert.cfg
>>>> and /var/lib/pki/pki-tomcat/ca/profiles/ca/caServerCert.cfg are present
>>>> and are identical.   The pki-ca package
>>>> passes rpm -V as well.   Are there any other troubleshooting steps we can
>>>> take?
>>> 
>>> Can you please check if the profile is available in the LDAP trees:
>>> 
>>> # ldapsearch -LLLx -D "cn=Directory Manager" -W -b 
>>> cn=certprofiles,cn=ca,$suffix
>> 
>> dn: cn=certprofiles,cn=ca,$suffix
>> objectClass: nsContainer
>> objectClass: top
>> cn: certprofiles
>> 
>>> # ldapsearch -LLLx -D "cn=Directory Manager" -W -b 
>>> ou=certificateProfiles,ou=ca,o=ipaca
>> 
>> dn: ou=certificateProfiles,ou=ca,o=ipaca
>> objectClass: top
>> objectClass: organizationalUnit
>> ou: certificateProfiles
>> 
>>> 
>>> If this is the case, please check if the profile is accessable by the
>>> host:
>>> 
>>> # kinit -kt /etc/krb5.keytab; klist; ipa certprofile-show caIPAserviceCert
>>> 
>> 
>> ipa: ERROR: caIPAserviceCert: Certificate Profile not found
>> 
>>> I either suspect that the profiles have not been properly migrated to
>>> the LDAP tree or that some ACIs are missing to allow access to the
>>> profiles.
>>> 
>> 
>> I suspect you’re right..I ran these same commands on a reference system and 
>> there was
>> a lot more output in the ldapsearches and the ipa certprofile-show command 
>> came back with
>> Profile ID: caIPAserviceCert
>> Profile description: Standard profile for network services
>> Store issued certificates: TRUE
> 
> Yes, this is a known issue which has been fixed in the most recent
> FreeIPA releases 4.2.4 and 4.3.1. 
> I would recommend to upgrade your system to one of those releases. If this is 
> not feasible, I can send you instructions how to fix the issue manually.
> 

It’s currently at 4.2.0-15.el7.centos.3..would the update 
4.2.0-15.0.1.el7.centos.6 have the fix backported?  Also, should 
com.netscape.cmscore.profile be changed in 
/var/lib/pki/pki-tomcat/ca/conf/CS.cfg beforehand?

Thanks,

> Cheers,
> Thorsten
> 





"This message and any attachments may contain confidential information. If you
have received this  message in error, any use or distribution is prohibited. 
Please notify us by reply e-mail if you have mistakenly received this message,
and immediately and permanently delete it and any attachments. Thank you."

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

2016-03-28 Thread Timothy Geier 

> On Mar 28, 2016, at 12:53 PM, Thorsten Scherf <tsch...@redhat.com> wrote:
> 
> On [Sat, 26.03.2016 03:26], Timothy Geier wrote:
>>  To follow up on this issue, we haven’t been able to get any further since
>>  last month due to the missing caServerCert profile..the configuration
>>  files /usr/share/pki/ca/profiles/ca/caServerCert.cfg
>>  and /var/lib/pki/pki-tomcat/ca/profiles/ca/caServerCert.cfg are present
>>  and are identical.   The pki-ca package
>>  passes rpm -V as well.   Are there any other troubleshooting steps we can
>>  take?
> 
> Can you please check if the profile is available in the LDAP trees:
> 
> # ldapsearch -LLLx -D "cn=Directory Manager" -W -b 
> cn=certprofiles,cn=ca,$suffix

dn: cn=certprofiles,cn=ca,$suffix
objectClass: nsContainer
objectClass: top
cn: certprofiles

> # ldapsearch -LLLx -D "cn=Directory Manager" -W -b 
> ou=certificateProfiles,ou=ca,o=ipaca

dn: ou=certificateProfiles,ou=ca,o=ipaca
objectClass: top
objectClass: organizationalUnit
ou: certificateProfiles

> 
> If this is the case, please check if the profile is accessable by the
> host:
> 
> # kinit -kt /etc/krb5.keytab; klist; ipa certprofile-show caIPAserviceCert
> 

ipa: ERROR: caIPAserviceCert: Certificate Profile not found

> I either suspect that the profiles have not been properly migrated to
> the LDAP tree or that some ACIs are missing to allow access to the
> profiles.
> 

I suspect you’re right..I ran these same commands on a reference system and 
there was
a lot more output in the ldapsearches and the ipa certprofile-show command came 
back with
  Profile ID: caIPAserviceCert
  Profile description: Standard profile for network services
  Store issued certificates: TRUE

Thanks,

> Cheers,
> Thorsten
> 
> -- 
> Manage your subscription for the Freeipa-users mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go to http://freeipa.org for more info on the project





"This message and any attachments may contain confidential information. If you
have received this  message in error, any use or distribution is prohibited. 
Please notify us by reply e-mail if you have mistakenly received this message,
and immediately and permanently delete it and any attachments. Thank you."

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

2016-03-25 Thread Timothy Geier 

On Feb 28, 2016, at 2:15 AM, Timothy Geier 
<tge...@accertify.com<mailto:tge...@accertify.com>> wrote:


On Feb 23, 2016, at 4:22 AM, Ludwig Krispenz 
<lkris...@redhat.com<mailto:lkris...@redhat.com>> wrote:


On 02/22/2016 11:51 PM, Timothy Geier wrote:

What’s the established procedure to start a 389 instance without any 
replication agreements enabled?  The only thing that seemed close on google 
(http://directory.fedoraproject.org/docs/389ds/howto/howto-fix-and-reset-time-skew.html)
 seems risky and couldn’t be done
trivially in a production environment.
no, this is about how to get out of problems when replication could no longer 
synchronize its csn time generation, either by too many accumulate time drifts 
o playing with system time, hope you don't have to go thru this.

Enabling disabling a replication agreement can be done by setting the 
configuration parameter:

look for replication agreements (entries with 
objectclass=nsDS5ReplicationAgreement) and set
nsds5ReplicaEnabled: off

you can do this with an ldapmodify when the server is running or by editing 
/etc/dirsrv/slapd-/dse.ldif when teh server is stopped

Thanks for the procedure..the good news is this worked quite well in making 
sure that 389 didn’t crash immediately after startup.  The bad news is that the 
certificates still didn’t renew due to

Server at 
"http://master_server:8080/ca/ee/ca/profileSubmit<https://mail.accertify.com/owa/redir.aspx?REF=hBo37W2qnlmUfAeXTrhGw6WdavZzsQoMPQ85UuuxxhZLgX6LCUDTCAFodHRwOi8vbWFzdGVyX3NlcnZlcjo4MDgwL2NhL2VlL2NhL3Byb2ZpbGVTdWJtaXQ.>"
 replied: Profile caServerCert Not Found

which was the same error in getcert list I saw that one time 389 didn’t crash 
right away.  At least now this can be further troubleshooted without worrying 
about 389.



To follow up on this issue, we haven’t been able to get any further since last 
month due to the missing caServerCert profile..the configuration files 
/usr/share/pki/ca/profiles/ca/caServerCert.cfg and 
/var/lib/pki/pki-tomcat/ca/profiles/ca/caServerCert.cfg are present and are 
identical.   The pki-ca package
passes rpm -V as well.   Are there any other troubleshooting steps we can take?








"This message and any attachments may contain confidential information. If you
have received this  message in error, any use or distribution is prohibited. 
Please notify us by reply e-mail if you have mistakenly received this message,
and immediately and permanently delete it and any attachments. Thank you."-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

2016-02-22 Thread Timothy Geier 

On Feb 22, 2016, at 9:21 AM, Ludwig Krispenz 
<lkris...@redhat.com<mailto:lkris...@redhat.com>> wrote:

The crash is an abort because of a failed assertion in  the kerberos code

Thread 1 (Thread 0x7fa7d4c88700 (LWP 3125)):
#0  0x7fa7e6ace5f7 in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x7fa7e6acfce8 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x7fa7e6ac7566 in __assert_fail_base () from /lib64/libc.so.6
No symbol table info available.
#3  0x7fa7e6ac7612 in __assert_fail () from /lib64/libc.so.6
No symbol table info available.
#4  0x7fa7e8d71b83 in k5_mutex_lock.part.1 () from /lib64/libkrb5.so.3
No symbol table info available.
#5  0x7fa7e8d7bda1 in k5_cc_mutex_lock () from /lib64/libkrb5.so.3
No symbol table info available.
#6  0x7fa7e8d851bf in krb5_mcc_store () from /lib64/libkrb5.so.3
No symbol table info available.
#7  0x7fa7e8d88070 in krb5_cc_store_cred () from /lib64/libkrb5.so.3
No symbol table info available.
#8  0x7fa7e8d98be3 in complete () from /lib64/libkrb5.so.3
No symbol table info available.
#9  0x7fa7e8d99ba1 in krb5_tkt_creds_step () from /lib64/libkrb5.so.3
No symbol table info available.
#10 0x7fa7e8d9a637 in krb5_tkt_creds_get () from /lib64/libkrb5.so.3
No symbol table info available.
#11 0x7fa7e8d9a75c in krb5_get_credentials () from /lib64/libkrb5.so.3
No symbol table info available.
#12 0x7fa7e0f6736a in krb5_gss_init_sec_context_ext () from 
/lib64/libgssapi_krb5.so.2
No symbol table info available.
#13 0x7fa7e0f67c97 in krb5_gss_init_sec_context () from 
/lib64/libgssapi_krb5.so.2
No symbol table info available.
#14 0x7fa7e0f516ab in gss_init_sec_context () from 
/lib64/libgssapi_krb5.so.2
No symbol table info available.
#15 0x7fa7e118ac44 in gssapi_client_mech_step () from 
/usr/lib64/sasl2/libgssapiv2.so
No symbol table info available.
#16 0x7fa7e72847f5 in sasl_client_step () from /lib64/libsasl2.so.3
No symbol table info available.
#17 0x7fa7e7284b76 in sasl_client_start () from /lib64/libsasl2.so.3
No symbol table info available.
#18 0x7fa7e8475e73 in ldap_int_sasl_bind () from /lib64/libldap_r-2.4.so.2
No symbol table info available.
#19 0x7fa7e8479492 in ldap_sasl_interactive_bind () from 
/lib64/libldap_r-2.4.so.2
No symbol table info available.
#20 0x7fa7e84796bd in ldap_sasl_interactive_bind_s () from 
/lib64/libldap_r-2.4.so.2

Directory server tries to open a replication connetion usig GSSAPI. I don't 
know which assertion fails in krb, but
- could you try with the replication agreement disabled ?
- Rob, you have been discussing renewals of keytabs, would we have to renew the 
ds.keytab ?


What’s the established procedure to start a 389 instance without any 
replication agreements enabled?  The only thing that seemed close on google 
(http://directory.fedoraproject.org/docs/389ds/howto/howto-fix-and-reset-time-skew.html)
 seems risky and couldn’t be done
trivially in a production environment.

Thanks,


On 02/20/2016 07:57 AM, Timothy Geier wrote:

To update this thread..timeshifting to before the certs expired (Feb.
2) brings up the interesting issue of pki-tomcatd (sometimes) starting
while slapd crashes almost immediately after ipactl start is run..if it
doesn't crash right away, pki-tomcatd will start, but since slapd will
then go down right afterwards, the certs cannot be renewed.  After
returning to real time, slapd is just fine along with everything else
aside from pki-tomcatd.

A core dump is attached.


"This message and any attachments may contain confidential information. If you
have received this  message in error, any use or distribution is prohibited.
Please notify us by reply e-mail if you have mistakenly received this message,
and immediately and permanently delete it and any attachments. Thank you."



--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project





"This message and any attachments may contain confidential information. If you
have received this  message in error, any use or distribution is prohibited. 
Please notify us by reply e-mail if you have mistakenly received this message,
and immediately and permanently delete it and any attachments. Thank you."-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

2016-02-10 Thread Timothy Geier 

On Feb 10, 2016, at 3:01 AM, Rob Crittenden 
> wrote:

[09/Feb/2016:12:55:41 -0600] conn=109598 fd=287 slot=287 SSL connection from 
master_ip to master_ip
[09/Feb/2016:12:55:41 -0600] conn=109597 op=0 EXT oid="1.3.6.1.4.1.1466.20037" 
name="startTLS"
[09/Feb/2016:12:55:41 -0600] conn=109597 op=0 RESULT err=0 tag=120 nentries=0 
etime=0
[09/Feb/2016:12:55:41 -0600] conn=109598 Netscape Portable Runtime error -8181 
(Peer's Certificate has expired.); unauthenticated client CN=CA 
Subsystem,O=XXX.NET; issuer CN=Certificate 
Authority,O=XXX.NET
[09/Feb/2016:12:55:41 -0600] conn=109598 op=-1 fd=287 closed - Peer's 
Certificate has expired.


Ok, right. The subsystem cert expired on Feb 1 so you'd have to back at least 
that far in time to do the renewals.


There are a few entries in ou=People,o=ipaca that need to reflect the current 
state of certificates as well.

Ah, right, just realized that that’s a base that can looked up separately in 
LDAP..is there anything in particular to look for in there?




All of the host keytabs on all of the IPA servers are correct..are there any 
other keytabs to check?

No, just /etc/krb5.keytab. I think you should focus on getting the CA subsystem 
certs renewed and then we can look at the other things. So I'd go back in time 
to Jan 30 or so and just restart certmonger.

After doing so, certmonger appears to run smoothly and goes from SUBMITTING to 
MONITORING but the expiration date on all of the certs stays the same. (It’s 
the same result if ipa-getcert resubmit is run against all of the request 
IDs..quite perplexing)

If we do a total shutdown/rewind/restart, getcert list produces the following 
for these 3 certs during the time shift after the restart:

Request ID '20160209194022':
status: CA_UNREACHABLE
ca-error: Internal error
stuck: no
key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=XXX.NET
subject: CN=CA Audit,O=XXX.NET
expires: 2016-02-01 19:46:48 UTC
key usage: digitalSignature,nonRepudiation
pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert 
"auditSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160209194023':
status: CA_UNREACHABLE
ca-error: Internal error
stuck: no
key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert 
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=XXX.NET
subject: CN=OCSP Subsystem,O=XXX.NET
expires: 2016-02-01 19:46:47 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
eku: id-kp-OCSPSigning
pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert 
"ocspSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160209194024':
status: CA_UNREACHABLE
ca-error: Internal error
stuck: no
key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert 
cert-pki-ca',token='NSS Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert 
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=XXX.NET
subject: CN=CA Subsystem,O=XXX.NET
expires: 2016-02-01 19:46:47 UTC
key usage: 
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert 
"subsystemCert cert-pki-ca"
track: yes
auto-renew: yes

The most puzzling thing (in my opinion) about this issue is that timeshifting 
doesn’t seem to make any difference at all; pki-tomcatd still doesn’t start 
cleanly (the log entries are very similar) even though in theory those certs 
are no longer expired at that point..it seems as if something else is also the 
issue.

Your ongoing assistance with this matter is much appreciated.



rob





"This message and any

Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

2016-02-09 Thread Timothy Geier 

> On Feb 9, 2016, at 2:58 AM, Rob Crittenden <rcrit...@redhat.com> wrote:
> 
> Timothy Geier wrote:
>> 
>> 
>> The debug log has a lot of instances of:
>> 
>> Could not connect to LDAP server host xxx. port 636 Error
>> netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1)
>> Internal Database Error encountered: Could not connect to LDAP server
>> host xxx. port 636 Error netscape.ldap.LDAPException: IO Error
>> creating JSS SSL Socket (-1)
> 
> The 389-ds access log may show connection attempts and you might be able to 
> correlate from there.
> 

[09/Feb/2016:12:55:41 -0600] conn=109598 fd=287 slot=287 SSL connection from 
master_ip to master_ip
[09/Feb/2016:12:55:41 -0600] conn=109597 op=0 EXT oid="1.3.6.1.4.1.1466.20037" 
name="startTLS"
[09/Feb/2016:12:55:41 -0600] conn=109597 op=0 RESULT err=0 tag=120 nentries=0 
etime=0
[09/Feb/2016:12:55:41 -0600] conn=109598 Netscape Portable Runtime error -8181 
(Peer's Certificate has expired.); unauthenticated client CN=CA 
Subsystem,O=XXX.NET; issuer CN=Certificate Authority,O=XXX.NET
[09/Feb/2016:12:55:41 -0600] conn=109598 op=-1 fd=287 closed - Peer's 
Certificate has expired.



> 
> Some data is also stored in CS.cfg so you might ensure that the certificates 
> stored there are correct as well.
> 

CS.cfg is correct/restored from backup.

> There are a few entries in ou=People,o=ipaca that need to reflect the current 
> state of certificates as well.
> 

While looking in cn=config for any o=ipaca entries, the following popped up..it 
looks like most of the old replicas and the old master hostnames weren’t 
properly cleaned up and there’s still RUVs for them..nsDS5ReplicaHost and Port 
also seem to be improperly set:

cn=cloneAgreement1-current-master.X.net-pki-tomcat,cn=replica,cn=o\\3Dipaca,cn=mapping
 tree,cn=config
cn: cloneAgreement1-current-master.X.net-pki-tomcat
description: cloneAgreement1-current-master.X.net-pki-tomcat
nsDS5ReplicaBindDN: cn=Replication Manager 
masterAgreement1-current-master.X.net-pki-tomcat,ou=csusers,cn=config
nsDS5ReplicaBindMethod: Simple
nsDS5ReplicaHost: old-master.X.net
nsDS5ReplicaPort: 7389
nsDS5ReplicaRoot: o=ipaca
nsDS5ReplicaTransportInfo: TLS
nsds50ruv: {replicageneration} 5304e0780060
nsds50ruv: {replica 96 ldap://old-master.X.net:7389} 
5304e0840060 561f3e8f00050060
nsds50ruv: {replica 81 ldap://current-master.X.net:389} 
561f361b0051 561f369c0051
nsds50ruv: {replica 86 ldap://old-replica3.X.net:7389} 
53f7b70d0056 5616ad6400090056
nsds50ruv: {replica 91 ldap://old-replica2.X.net:7389} 
53a841e5005b 5616ad5f0005005b
nsds50ruv: {replica 97 ldap://old-replica2.X.net:7389} 
5304e0800061 539b4f9600010061
nsruvReplicaLastModified: {replica 96 ldap://old-master.X.net:7389} 

nsruvReplicaLastModified: {replica 81 ldap://current-master.X.net:389} 

nsruvReplicaLastModified: {replica 86 ldap://old-replica3.X.net:7389} 

nsruvReplicaLastModified: {replica 91 ldap://old-replica2.X.net:7389} 

nsruvReplicaLastModified: {replica 97 ldap://old-replica2.X.net:7389} 

objectClass: top
objectClass: nsds5replicationagreement
nsds5replicareapactive: 0
nsds5replicaLastUpdateStart: 1970010100Z
nsds5replicaLastUpdateEnd: 1970010100Z
nsds5replicaChangesSentSinceStartup:
nsds5replicaLastUpdateStatus: -1 Unable to acquire replicaLDAP error: Can't 
contact LDAP server
nsds5replicaUpdateInProgress: FALSE
nsds5replicaLastInitStart: 1970010100Z
nsds5replicaLastInitEnd: 1970010100Z

cn=replica,cn=o\\3Dipaca,cn=mapping tree,cn=config
cn: replica
nsDS5Flags: 1
nsDS5ReplicaBindDN: cn=Replication Manager 
cloneAgreement1-current-master.X.net-pki-tomcat,ou=csusers,cn=config
nsDS5ReplicaId: 81
nsDS5ReplicaName: 56db714e-72e411e5-87dbe691-e12b51f2
nsDS5ReplicaRoot: o=ipaca
nsDS5ReplicaType: 3
nsState:: UQDkWbdWwCYBAA==
objectClass: top
objectClass: nsDS5Replica
objectClass: extensibleobject
nsds5ReplicaChangeCount: 11463
nsds5replicareapactive: 0

ou=People doesn’t seem to exist in either the main tree or cn=config.

>> 
>>> You mentioned privately that you renamed the IPA host. This is
>>> probably what broke half of the renewals. The hosts and keytabs and
>>> many entries in IPA have the hostname baked in so you can't simply
>>> rename the host.
>>> 
>> 
>> Technically, the host wasn’t renamed; a new CentOS 7 host was added to
>> the existing IPA cluster that had 4 hosts (1 master CA) at CentOS 6
>> (using the documentation at
>> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identit

Re: [Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

2016-02-08 Thread Timothy Geier 

On Feb 8, 2016, at 4:28 AM, Rob Crittenden 
<rcrit...@redhat.com<mailto:rcrit...@redhat.com>> wrote:

Timothy Geier wrote:
Greetings all,

For the record,this is a CentOS 7.2 box with all current patches. 
(ipa-server-4.2.0-15.el7.centos.3.x86_64, etc.)

The situation is that pki-tomcatd on the lone CA server in our IPA cluster 
refuses to start cleanly.  The issues started earlier this week after the certs
subsystemCert, ocspSigningCert, and auditSigningCert all simultaneously expired 
without warning; apparently, certmonger failed to renew them automatically.  We
attempted timeshifting and following instructions for what appeared to be 
similar issues, but nothing at all has worked.

Today, we attempted removing the certificates in question (of course, the files 
in /etc/pki/pki-tomcat/alias were backed up beforehand) and using certutil to 
issue new  certificates.   This process worked but pki-tomcatd is still 
refusing to start.  We can get IPA to run on this server by manually starting 
pki-tomcatd, running ipactl start, and then ctrl-c’ing it when it gets to 
"Starting pki-tomcatd" but this is not a tenable long-term solution.

Relevant log entries/information:

/var/log/pki/pki-tomcat/ca/debug:
Could not connect to LDAP server host 
ipa01.X.net<http://ipa01.X.net> port 636 Error 
netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1)
Internal Database Error encountered: Could not connect to LDAP server host 
ipa01.X.net<http://ipa01.X.net> port 636 Error 
netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1)
Internal Database Error encountered: Could not connect to LDAP server host 
ipa01.X.net<http://ipa01.X.net> port 636 Error 
netscape.ldap.LDAPException: Authentication failed (49)

/var/log/pki/pki-tomcat/localhost.2016-02-04.log:
org.apache.catalina.core.StandardContext loadOnStartup
SEVERE: Servlet /ca threw load() exception
java.lang.NullPointerException

# getcert list:

Number of certificates and requests being tracked: 8.
Request ID '20151015022737':
status: MONITORING
ca-error: Error setting up ccache for "host" service on client using default 
keytab: Generic error (see e-text).
stuck: no
key pair storage: 
type=NSSDB,location='/etc/dirsrv/slapd-X-NET',nickname='Server-Cert',token='NSS
 Certificate DB',pinfile='/etc/dirsrv/slapd-X-NET/pwdfile.txt'
expires: 2017-10-15 02:09:06 UTC
track: yes
auto-renew: yes
Request ID '20151015022949':
status: MONITORING
ca-error: Error setting up ccache for "host" service on client using default 
keytab: Generic error (see e-text).
stuck: no
key pair storage: 
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS 
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: 
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS 
Certificate DB'
expires: 2017-10-15 02:09:10 UTC
track: yes
auto-renew: yes
Request ID '20160127202548':
status: MONITORING
stuck: no
key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert 
cert-pki-ca',token='NSS Certificate DB'
expires: 2034-02-11 19:46:43 UTC
track: yes
auto-renew: yes
Request ID '20160127202549':
status: MONITORING
stuck: no
key pair storage: 
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS 
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: 
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS 
Certificate DB'
expires: 2017-12-25 04:27:49 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
track: yes
auto-renew: yes
Request ID '20160127202550':
status: MONITORING
ca-error: Server at "http://ipa01.X.net:8080/ca/ee/ca/profileSubmit; 
replied: Profile caServerCert Not Found
stuck: no
key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert 
cert-pki-ca',token='NSS Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert 
cert-pki-ca',token='NSS Certificate DB'
expires: 2017-10-04 02:28:53 UTC
track: yes
auto-renew: yes
Request ID '20160204165453':
status: MONITORING
stuck: no
key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
cert-pki-ca',token='NSS Certificate DB'
expires: 2016-05-04 16:40:23 UTC
track: yes
auto-renew: yes
Request ID '20160204170246':
status: MONITORING
stuck: no
key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspS

[Freeipa-users] IPA 4.2: pki-tomcatd in terrible shape

2016-02-04 Thread Timothy Geier 
Greetings all,

For the record,this is a CentOS 7.2 box with all current patches. 
(ipa-server-4.2.0-15.el7.centos.3.x86_64, etc.)

The situation is that pki-tomcatd on the lone CA server in our IPA cluster 
refuses to start cleanly.  The issues started earlier this week after the certs
subsystemCert, ocspSigningCert, and auditSigningCert all simultaneously expired 
without warning; apparently, certmonger failed to renew them automatically.  We
attempted timeshifting and following instructions for what appeared to be 
similar issues, but nothing at all has worked.  

Today, we attempted removing the certificates in question (of course, the files 
in /etc/pki/pki-tomcat/alias were backed up beforehand) and using certutil to 
issue new  certificates.   This process worked but pki-tomcatd is still 
refusing to start.  We can get IPA to run on this server by manually starting 
pki-tomcatd, running ipactl start, and then ctrl-c’ing it when it gets to 
"Starting pki-tomcatd" but this is not a tenable long-term solution.

Relevant log entries/information:

/var/log/pki/pki-tomcat/ca/debug:
Could not connect to LDAP server host ipa01.X.net port 636 Error 
netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1)
Internal Database Error encountered: Could not connect to LDAP server host 
ipa01.X.net port 636 Error netscape.ldap.LDAPException: IO Error 
creating JSS SSL Socket (-1)
Internal Database Error encountered: Could not connect to LDAP server host 
ipa01.X.net port 636 Error netscape.ldap.LDAPException: Authentication 
failed (49)

/var/log/pki/pki-tomcat/localhost.2016-02-04.log:
org.apache.catalina.core.StandardContext loadOnStartup
SEVERE: Servlet /ca threw load() exception
java.lang.NullPointerException

# getcert list:

Number of certificates and requests being tracked: 8.
Request ID '20151015022737':
status: MONITORING
ca-error: Error setting up ccache for "host" service on client using 
default keytab: Generic error (see e-text).
stuck: no
key pair storage: 
type=NSSDB,location='/etc/dirsrv/slapd-X-NET',nickname='Server-Cert',token='NSS
 Certificate DB',pinfile='/etc/dirsrv/slapd-X-NET/pwdfile.txt'
expires: 2017-10-15 02:09:06 UTC
track: yes
auto-renew: yes
Request ID '20151015022949':
status: MONITORING
ca-error: Error setting up ccache for "host" service on client using 
default keytab: Generic error (see e-text).
stuck: no
key pair storage: 
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS 
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: 
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS 
Certificate DB'
expires: 2017-10-15 02:09:10 UTC
track: yes
auto-renew: yes
Request ID '20160127202548':
status: MONITORING
stuck: no
key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert 
cert-pki-ca',token='NSS Certificate DB'
expires: 2034-02-11 19:46:43 UTC
track: yes
auto-renew: yes
Request ID '20160127202549':
status: MONITORING
stuck: no
key pair storage: 
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS 
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: 
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS 
Certificate DB'
expires: 2017-12-25 04:27:49 UTC
key usage: 
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
track: yes
auto-renew: yes
Request ID '20160127202550':
status: MONITORING
ca-error: Server at 
"http://ipa01.X.net:8080/ca/ee/ca/profileSubmit; replied: Profile 
caServerCert Not Found
stuck: no
key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert 
cert-pki-ca',token='NSS Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert 
cert-pki-ca',token='NSS Certificate DB'
expires: 2017-10-04 02:28:53 UTC
track: yes
auto-renew: yes
Request ID '20160204165453':
status: MONITORING
stuck: no
key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
cert-pki-ca',token='NSS Certificate DB'
expires: 2016-05-04 16:40:23 UTC
track: yes
auto-renew: yes
Request ID '20160204170246':
status: MONITORING
stuck: no
key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert