Re: [Freeipa-users] Failure decoding Certificate Signing Request

2013-10-30 Thread Rob Crittenden

Thomson, Ryan wrote:

-Original Message-
From: freeipa-users-boun...@redhat.com [mailto:freeipa-users-
boun...@redhat.com] On Behalf Of Thomson, Ryan
Sent: Friday, October 25, 2013 11:17 AM
To: Rob Crittenden; freeipa-users@redhat.com
Subject: Re: [Freeipa-users] Failure decoding Certificate Signing Request


[snip]


I'm not sure what to make of this.


This is just more confirmation that the IPA framework is trying to
initialize NSS for some reason. It should never do this which is why
it is failing so spectacularly.

Can you provide nss.conf and ipa.conf from /etc/httpd/conf.d?

Who owns and what are the permissions of /etc/httpd/alias/*.db?

thanks

rob


[snip]

After further troubleshooting and trying too many crazy things, I was finally able to 
"hack" FreeIPA into renewing the certificates. Behold the terrible hack, it is 
quite awful and certainly not an actual fix.

During my troubleshooting, I kept coming back to the getpass EOFError:

[Sat Sep 28 03:48:17 2013] [error] /usr/lib64/python2.6/getpass.py:83: 
GetPassWarning: Can not control echo on the terminal.
[Sat Sep 28 03:48:17 2013] [error]   passwd = fallback_getpass(prompt, stream)
[Sat Sep 28 03:48:17 2013] [error] Warning: Password input may be echoed.
[Sat Sep 28 03:48:17 2013] [error] Enter password for internal:
[Sat Sep 28 03:48:17 2013] [error] exception in PK11 password callback
[Sat Sep 28 03:48:17 2013] [error] Traceback (most recent call last):
[Sat Sep 28 03:48:18 2013] [error]   File 
"/usr/lib/python2.6/site-packages/ipapython/nsslib.py", line 230, in 
password_callback
[Sat Sep 28 03:48:18 2013] [error] return getpass.getpass("Enter password for 
%s: " % slot.token_name);
[Sat Sep 28 03:48:18 2013] [error]   File "/usr/lib64/python2.6/getpass.py", 
line 83, in unix_getpass
[Sat Sep 28 03:48:18 2013] [error] passwd = fallback_getpass(prompt, stream)
[Sat Sep 28 03:48:18 2013] [error]   File "/usr/lib64/python2.6/getpass.py", 
line 118, in fallback_getpass
[Sat Sep 28 03:48:18 2013] [error] return _raw_input(prompt, stream)
[Sat Sep 28 03:48:18 2013] [error]   File "/usr/lib64/python2.6/getpass.py", 
line 135, in _raw_input
[Sat Sep 28 03:48:18 2013] [error] raise EOFError
[Sat Sep 28 03:48:18 2013] [error] EOFError

So eventually I found my way into /usr/lib/python2.6/site-packages/ipapython/nsslib.py and the 
getpass password_callback function. Instead of allowing the password_callback() to return 
"getpass.getpass("Enter password for %s: " % slot.token_name);", which was 
rasing the EOFError exception, I simply returned the text string from 
/etc/httpd/alias/password.conf!

 From what little I know about python and coding in general, it seems that 
getpass was trying to read the NSS DB password from somewhere but was reading 
nothing instead and thus raising EOFError.

I'm not sure what to do about a permanent fix as maintaining the NSS DB 
password in a source file does seems rather insecure. Given the above, what 
does a permanent fix actually look like? Is the getpass callback function 
broken or is my environment broken?


Right, and removing it will cause other cert requests to fail in the 
same way. It will also get lost upon the next update of the ipa-python 
package.


The root of the problem is that for some unknown reason the NSS database 
isn't already initialized. I've been unable to reproduce that here so 
don't really have any idea what the problem could be.


A slightly more secure workaround might be to set an empty password on 
the NSS database in /etc/httpd/alias. SELinux and FS permissions will 
protect the key material but a password won't be required any more. This 
would mean some other configuration changes, and I don't know that we've 
tested this specifically, but it would probably work. Certainly the sort 
of thing where you back up a lot of things first.


rob

___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users


Re: [Freeipa-users] Failure decoding Certificate Signing Request

2013-10-29 Thread Thomson, Ryan
> -Original Message-
> From: freeipa-users-boun...@redhat.com [mailto:freeipa-users-
> boun...@redhat.com] On Behalf Of Thomson, Ryan
> Sent: Friday, October 25, 2013 11:17 AM
> To: Rob Crittenden; freeipa-users@redhat.com
> Subject: Re: [Freeipa-users] Failure decoding Certificate Signing Request

[snip]

> > > I'm not sure what to make of this.
> >
> > This is just more confirmation that the IPA framework is trying to
> > initialize NSS for some reason. It should never do this which is why
> > it is failing so spectacularly.
> >
> > Can you provide nss.conf and ipa.conf from /etc/httpd/conf.d?
> >
> > Who owns and what are the permissions of /etc/httpd/alias/*.db?
> >
> > thanks
> >
> > rob

[snip]

After further troubleshooting and trying too many crazy things, I was finally 
able to "hack" FreeIPA into renewing the certificates. Behold the terrible 
hack, it is quite awful and certainly not an actual fix.

During my troubleshooting, I kept coming back to the getpass EOFError:

[Sat Sep 28 03:48:17 2013] [error] /usr/lib64/python2.6/getpass.py:83: 
GetPassWarning: Can not control echo on the terminal.
[Sat Sep 28 03:48:17 2013] [error]   passwd = fallback_getpass(prompt, stream)
[Sat Sep 28 03:48:17 2013] [error] Warning: Password input may be echoed.
[Sat Sep 28 03:48:17 2013] [error] Enter password for internal: 
[Sat Sep 28 03:48:17 2013] [error] exception in PK11 password callback
[Sat Sep 28 03:48:17 2013] [error] Traceback (most recent call last):
[Sat Sep 28 03:48:18 2013] [error]   File 
"/usr/lib/python2.6/site-packages/ipapython/nsslib.py", line 230, in 
password_callback
[Sat Sep 28 03:48:18 2013] [error] return getpass.getpass("Enter password 
for %s: " % slot.token_name);
[Sat Sep 28 03:48:18 2013] [error]   File "/usr/lib64/python2.6/getpass.py", 
line 83, in unix_getpass
[Sat Sep 28 03:48:18 2013] [error] passwd = fallback_getpass(prompt, stream)
[Sat Sep 28 03:48:18 2013] [error]   File "/usr/lib64/python2.6/getpass.py", 
line 118, in fallback_getpass
[Sat Sep 28 03:48:18 2013] [error] return _raw_input(prompt, stream)
[Sat Sep 28 03:48:18 2013] [error]   File "/usr/lib64/python2.6/getpass.py", 
line 135, in _raw_input
[Sat Sep 28 03:48:18 2013] [error] raise EOFError
[Sat Sep 28 03:48:18 2013] [error] EOFError

So eventually I found my way into 
/usr/lib/python2.6/site-packages/ipapython/nsslib.py and the getpass 
password_callback function. Instead of allowing the password_callback() to 
return "getpass.getpass("Enter password for %s: " % slot.token_name);", which 
was rasing the EOFError exception, I simply returned the text string from 
/etc/httpd/alias/password.conf! 

>From what little I know about python and coding in general, it seems that 
>getpass was trying to read the NSS DB password from somewhere but was reading 
>nothing instead and thus raising EOFError.

I'm not sure what to do about a permanent fix as maintaining the NSS DB 
password in a source file does seems rather insecure. Given the above, what 
does a permanent fix actually look like? Is the getpass callback function 
broken or is my environment broken?

--Ryan

___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users


Re: [Freeipa-users] Failure decoding Certificate Signing Request

2013-10-25 Thread Thomson, Ryan
> -Original Message-
> From: Rob Crittenden [mailto:rcrit...@redhat.com]
> Sent: Friday, October 25, 2013 10:54 AM
> To: Thomson, Ryan; freeipa-users@redhat.com
> Subject: Re: [Freeipa-users] Failure decoding Certificate Signing Request
> 
> Thomson, Ryan wrote:
> >> -Original Message-
> >> From: Rob Crittenden [mailto:rcrit...@redhat.com]
> >> Sent: Thursday, October 24, 2013 11:41 AM
> >> To: Thomson, Ryan; freeipa-users@redhat.com
> >> Subject: Re: [Freeipa-users] Failure decoding Certificate Signing
> >> Request
> >>
> >> Thomson, Ryan wrote:
> >>>> -Original Message-
> >>>> From: Rob Crittenden [mailto:rcrit...@redhat.com]
> >>>> Sent: Wednesday, October 23, 2013 6:58 PM
> >>>> To: Thomson, Ryan; freeipa-users@redhat.com
> >>>> Subject: Re: [Freeipa-users] Failure decoding Certificate Signing
> >>>> Request
> >>>>
> >>>> I think this still points to NSS not being initialized. The way we
> >>>> currently use NSS in the server is Apache fires things up using
> >>>> mod_nss, then because we are a child of Apache via mod_wsgi, we
> >>>> inherit the open NSS database in /etc/httpd/alias. This gives us
> >>>> the CA cert and the client cert we need in order to talk to dogtag.
> >>>>
> >>>> What I thought, and the excellent debugging above confirms, is that
> >>>> at some point the NSS database is being shut down. At some point we
> >>>> need to do some crypto and try to initialize it ourselves to no
> >>>> avail. We shouldn't ever need to do it in the server and thus don't
> >>>> have access to PINs and such because we don't need them. We do
> >>>> initialize things from time to time on the client side but we tend
> >>>> to do a database-less initialization (nss_init_nodb()).
> >>>>
> >>>> I'm not really sure what this tells us though. It would appear that
> >>>> SSL is working in Apache, because you are able to get far enough to
> >>>> make a request and have it fail. So the NSS database is still
> >>>> initialized in Apache, but for some reason the wsgi code doesn't
> >>>> seem to
> >> agree.
> >>>>
> >>>> Would it be possible for you to stop and restart Apache and run
> >>>> some simple IPA command like ipa user-show admin (and let me know
> >>>> if it
> >> succeeds)?
> >>>> Then send me the error_log?
> >>>>
> >>>> If you are in SELinux enforcing mode it would also be helpful to
> >>>> check for any AVCs. Maybe we simply can't access the database.
> >>>>
> >>>> thanks
> >>>>
> >>>> rob
> >>>
> >>> I am able to stop/wait/start apache and then execute "ipa user-show
> >> admin" successfully.
> >>>
> >>
> >> Ok, let's try a couple more things.
> >>
> >> Can you set LogLevel debug in /etc/httpd/conf.d/nss.conf and restart
> >> Apache again? This may give us more information on what mod_nss is
> doing.
> >>
> >> Next, lets try a different cert command that should also invoke the
> >> NSS client within IPA:
> >>
> >> $ ipa cert-show 22
> >>
> >> Can you describe your environment? Do you have multiple IPA masters?
> >> Was this a new install at 3.0 or is it an upgrade from 2.2?
> >>
> >> rob
> >
> > The environment is simple: Single master, upgraded from 2.2.
> >
> > Output in /var/log/httpd/error_log after setting LogLevel to debug in
> /etc/httpd/conf.d/nss.conf and restarting apache:
> 
> [ snip ]
> 
> >
> > I'm not sure what to make of this.
> 
> This is just more confirmation that the IPA framework is trying to initialize
> NSS for some reason. It should never do this which is why it is failing so
> spectacularly.
> 
> Can you provide nss.conf and ipa.conf from /etc/httpd/conf.d?
> 
> Who owns and what are the permissions of /etc/httpd/alias/*.db?
> 
> thanks
> 
> rob


Thanks for sticking with me on this.

Below is the contents of the requested files and ownership/permissions of the 
apache NSS db. 

/etc/httpd/conf.d/ipa.conf:

#
# VERSION 10 - DO NOT REMOVE THIS LINE
#
# This file may be overwritten on upgrades.
#
# LoadModule auth_kerb_module modules/mod_auth_kerb.so

ProxyRequests Off


#We u

Re: [Freeipa-users] Failure decoding Certificate Signing Request

2013-10-25 Thread Rob Crittenden

Thomson, Ryan wrote:

-Original Message-
From: Rob Crittenden [mailto:rcrit...@redhat.com]
Sent: Thursday, October 24, 2013 11:41 AM
To: Thomson, Ryan; freeipa-users@redhat.com
Subject: Re: [Freeipa-users] Failure decoding Certificate Signing Request

Thomson, Ryan wrote:

-Original Message-
From: Rob Crittenden [mailto:rcrit...@redhat.com]
Sent: Wednesday, October 23, 2013 6:58 PM
To: Thomson, Ryan; freeipa-users@redhat.com
Subject: Re: [Freeipa-users] Failure decoding Certificate Signing
Request

I think this still points to NSS not being initialized. The way we
currently use NSS in the server is Apache fires things up using
mod_nss, then because we are a child of Apache via mod_wsgi, we
inherit the open NSS database in /etc/httpd/alias. This gives us the
CA cert and the client cert we need in order to talk to dogtag.

What I thought, and the excellent debugging above confirms, is that
at some point the NSS database is being shut down. At some point we
need to do some crypto and try to initialize it ourselves to no
avail. We shouldn't ever need to do it in the server and thus don't
have access to PINs and such because we don't need them. We do
initialize things from time to time on the client side but we tend to
do a database-less initialization (nss_init_nodb()).

I'm not really sure what this tells us though. It would appear that
SSL is working in Apache, because you are able to get far enough to
make a request and have it fail. So the NSS database is still
initialized in Apache, but for some reason the wsgi code doesn't seem to

agree.


Would it be possible for you to stop and restart Apache and run some
simple IPA command like ipa user-show admin (and let me know if it

succeeds)?

Then send me the error_log?

If you are in SELinux enforcing mode it would also be helpful to
check for any AVCs. Maybe we simply can't access the database.

thanks

rob


I am able to stop/wait/start apache and then execute "ipa user-show

admin" successfully.




Ok, let's try a couple more things.

Can you set LogLevel debug in /etc/httpd/conf.d/nss.conf and restart
Apache again? This may give us more information on what mod_nss is doing.

Next, lets try a different cert command that should also invoke the NSS client
within IPA:

$ ipa cert-show 22

Can you describe your environment? Do you have multiple IPA masters? Was
this a new install at 3.0 or is it an upgrade from 2.2?

rob


The environment is simple: Single master, upgraded from 2.2.

Output in /var/log/httpd/error_log after setting LogLevel to debug in 
/etc/httpd/conf.d/nss.conf and restarting apache:


[ snip ]



I'm not sure what to make of this.


This is just more confirmation that the IPA framework is trying to 
initialize NSS for some reason. It should never do this which is why it 
is failing so spectacularly.


Can you provide nss.conf and ipa.conf from /etc/httpd/conf.d?

Who owns and what are the permissions of /etc/httpd/alias/*.db?

thanks

rob

___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users


Re: [Freeipa-users] Failure decoding Certificate Signing Request

2013-10-24 Thread Rob Crittenden

Thomson, Ryan wrote:

-Original Message-
From: Rob Crittenden [mailto:rcrit...@redhat.com]
Sent: Wednesday, October 23, 2013 6:58 PM
To: Thomson, Ryan; freeipa-users@redhat.com
Subject: Re: [Freeipa-users] Failure decoding Certificate Signing Request

I think this still points to NSS not being initialized. The way we currently use
NSS in the server is Apache fires things up using mod_nss, then because we
are a child of Apache via mod_wsgi, we inherit the open NSS database in
/etc/httpd/alias. This gives us the CA cert and the client cert we need in
order to talk to dogtag.

What I thought, and the excellent debugging above confirms, is that at some
point the NSS database is being shut down. At some point we need to do
some crypto and try to initialize it ourselves to no avail. We shouldn't ever
need to do it in the server and thus don't have access to PINs and such
because we don't need them. We do initialize things from time to time on
the client side but we tend to do a database-less initialization
(nss_init_nodb()).

I'm not really sure what this tells us though. It would appear that SSL is
working in Apache, because you are able to get far enough to make a
request and have it fail. So the NSS database is still initialized in Apache, 
but
for some reason the wsgi code doesn't seem to agree.

Would it be possible for you to stop and restart Apache and run some simple
IPA command like ipa user-show admin (and let me know if it succeeds)?
Then send me the error_log?

If you are in SELinux enforcing mode it would also be helpful to check for any
AVCs. Maybe we simply can't access the database.

thanks

rob


I am able to stop/wait/start apache and then execute "ipa user-show admin" 
successfully.



Ok, let's try a couple more things.

Can you set LogLevel debug in /etc/httpd/conf.d/nss.conf and restart 
Apache again? This may give us more information on what mod_nss is doing.


Next, lets try a different cert command that should also invoke the NSS 
client within IPA:


$ ipa cert-show 22

Can you describe your environment? Do you have multiple IPA masters? Was 
this a new install at 3.0 or is it an upgrade from 2.2?


rob

___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users


Re: [Freeipa-users] Failure decoding Certificate Signing Request

2013-10-24 Thread Thomson, Ryan
> -Original Message-
> From: Rob Crittenden [mailto:rcrit...@redhat.com]
> Sent: Wednesday, October 23, 2013 6:58 PM
> To: Thomson, Ryan; freeipa-users@redhat.com
> Subject: Re: [Freeipa-users] Failure decoding Certificate Signing Request
> 
> I think this still points to NSS not being initialized. The way we currently 
> use
> NSS in the server is Apache fires things up using mod_nss, then because we
> are a child of Apache via mod_wsgi, we inherit the open NSS database in
> /etc/httpd/alias. This gives us the CA cert and the client cert we need in
> order to talk to dogtag.
> 
> What I thought, and the excellent debugging above confirms, is that at some
> point the NSS database is being shut down. At some point we need to do
> some crypto and try to initialize it ourselves to no avail. We shouldn't ever
> need to do it in the server and thus don't have access to PINs and such
> because we don't need them. We do initialize things from time to time on
> the client side but we tend to do a database-less initialization
> (nss_init_nodb()).
> 
> I'm not really sure what this tells us though. It would appear that SSL is
> working in Apache, because you are able to get far enough to make a
> request and have it fail. So the NSS database is still initialized in Apache, 
> but
> for some reason the wsgi code doesn't seem to agree.
> 
> Would it be possible for you to stop and restart Apache and run some simple
> IPA command like ipa user-show admin (and let me know if it succeeds)?
> Then send me the error_log?
> 
> If you are in SELinux enforcing mode it would also be helpful to check for any
> AVCs. Maybe we simply can't access the database.
> 
> thanks
> 
> rob

I am able to stop/wait/start apache and then execute "ipa user-show admin" 
successfully.

Relevant error log from /var/log/httpd/error_log:

[Mon Oct 07 18:36:51 2013] [error] ipa: DEBUG: WSGI wsgi_dispatch.__call__:
[Mon Oct 07 18:36:51 2013] [error] ipa: DEBUG: WSGI xmlserver.__call__:
[Mon Oct 07 18:36:51 2013] [error] ipa: DEBUG: Created connection context.ldap2
[Mon Oct 07 18:36:51 2013] [error] ipa: DEBUG: WSGI WSGIExecutioner.__call__:
[Mon Oct 07 18:36:51 2013] [error] ipa: DEBUG: raw: user_show(u'admin', 
rights=False, all=False, raw=False, version=u'2.46')
[Mon Oct 07 18:36:51 2013] [error] ipa: DEBUG: user_show(u'admin', 
rights=False, all=False, raw=False, version=u'2.46')
[Mon Oct 07 18:36:51 2013] [error] ipa: DEBUG: retrieving schema for 
SchemaCache url=ldapi://%2fvar%2frun%2fslapd-FMRI-UBC-CA.socket 
conn=
[Mon Oct 07 18:36:52 2013] [error] ipa: DEBUG: get_memberof: 
entry_dn=uid=admin,cn=users,cn=accounts,dc=fully,dc=qualified,dc=domain 
memberof=[ipapython.dn.DN('cn=admins,cn=groups,cn=accounts,dc=fully,dc=qualified,dc=domain'),
 ipapython.dn.DN('cn=replication 
administrators,cn=privileges,cn=pbac,dc=fully,dc=qualified,dc=domain'), 
ipapython.dn.DN('cn=add replication 
agreements,cn=permissions,cn=pbac,dc=fully,dc=qualified,dc=domain'), 
ipapython.dn.DN('cn=modify replication 
agreements,cn=permissions,cn=pbac,dc=fully,dc=qualified,dc=domain'), 
ipapython.dn.DN('cn=remove replication 
agreements,cn=permissions,cn=pbac,dc=fully,dc=qualified,dc=domain'), 
ipapython.dn.DN('cn=host 
enrollment,cn=privileges,cn=pbac,dc=fully,dc=qualified,dc=domain'), 
ipapython.dn.DN('cn=manage host 
keytab,cn=permissions,cn=pbac,dc=fully,dc=qualified,dc=domain'), 
ipapython.dn.DN('cn=enroll a 
host,cn=permissions,cn=pbac,dc=fully,dc=qualified,dc=domain'), 
ipapython.dn.DN('cn=add krbprincipalname to a host,cn=p!
 ermissions,cn=pbac,dc=fully,dc=qualified,dc=domain'), 
ipapython.dn.DN('cn=unlock user 
accounts,cn=permissions,cn=pbac,dc=fully,dc=qualified,dc=domain'), 
ipapython.dn.DN('cn=manage service 
keytab,cn=permissions,cn=pbac,dc=fully,dc=qualified,dc=domain'), 
ipapython.dn.DN('cn=trust 
admins,cn=groups,cn=accounts,dc=fully,dc=qualified,dc=domain')]
[Mon Oct 07 18:36:52 2013] [error] ipa: DEBUG: get_memberof: result 
direct=[ipapython.dn.DN('cn=admins,cn=groups,cn=accounts,dc=fully,dc=qualified,dc=domain'),
 ipapython.dn.DN('cn=trust 
admins,cn=groups,cn=accounts,dc=fully,dc=qualified,dc=domain')] 
indirect=[ipapython.dn.DN('cn=replication 
administrators,cn=privileges,cn=pbac,dc=fully,dc=qualified,dc=domain'), 
ipapython.dn.DN('cn=add replication 
agreements,cn=permissions,cn=pbac,dc=fully,dc=qualified,dc=domain'), 
ipapython.dn.DN('cn=modify replication 
agreements,cn=permissions,cn=pbac,dc=fully,dc=qualified,dc=domain'), 
ipapython.dn.DN('cn=remove replication 
agreements,cn=permissions,cn=pbac,dc=fully,dc=qualified,dc=domain'), 
ipapython.dn.DN('cn=h

Re: [Freeipa-users] Failure decoding Certificate Signing Request

2013-10-23 Thread Rob Crittenden

Thomson, Ryan wrote:

-Original Message-
From: Rob Crittenden [mailto:rcrit...@redhat.com]
Sent: Tuesday, October 22, 2013 7:13 PM
To: Thomson, Ryan; freeipa-users@redhat.com
Subject: Re: [Freeipa-users] Failure decoding Certificate Signing Request

Thomson, Ryan wrote:

-Original Message-
From: Rob Crittenden [mailto:rcrit...@redhat.com]
Sent: Tuesday, October 22, 2013 10:46 AM
To: Thomson, Ryan; freeipa-users@redhat.com
Subject: Re: [Freeipa-users] Failure decoding Certificate Signing
Request

Thomson, Ryan wrote:

Hi Rob,


There is some duplication in the error strings (ticket
https://fedorahosted.org/freeipa/ticket/3988). Did you add a number
prefix to yours, I see a 3 -in the error. If so, by my calculation,
this works out to be an NSPRError. It would be helpful to know what
exception is being raised, which we don't do.


I did prefix numbers to the various error strings.


Either way, if you could enhance each occurrence of 'Failure
decoding Certificate Signing Request' in /usr/lib/python*/site-
packages/ipalib/plugins/cert.py to something like:

except NSPEError, nsprerr:
raise  errors.CertificateOperationError(error=_('Failure
decoding Certificate Signing Request" %s') % nsprerr)

You'll need to restart the httpd process afterwards. This should
give us the real reason for the failure.


Done. The error I get now is:

Server failed request, will retry: 4301 (RPC failed at server.
Certificate

operation cannot be completed: Failure decoding Certificate Signing
Request" [Errno -8018] error (-8018) unknown).

Hmm, very strange indeed.

It should be using the NSS database initialized in mod_nss for
Apache, which should remain open and available for wsgi. It almost
seems like the database has been shut down.

Can you add a logging event to log the value of nss.nss_is_initialized()?

Have you done any configuration customization in Apache or mod_nss?

thanks

rob


The return value of nss.nss_is_initialized() is False when I resubmit the

(expired) certs through certmonger.

Ok, that is the core of the issue then. pkcs10.load_certificate() will 
initialize
NSS If it isn't already and I'm guessing that is failing and is the source of 
this
exception.


I did have a custom config for apache that configured a virtual host with

SSL. I have disabled that config and restarted httpd, resubmitted the certs to
certmonger but I still receive the same error. I will continue poking through
my apache / mod_nss config to see if anything stands out.

You're still using mod_nss though, right?

rob


I'm still using mod_nss.

I have discovered that I might be focusing on a symptom here rather than the core 
problem. If I restart httpd and then certmonger, the first error returned when certmonger 
tries to renew the certificates is not "Failure decoding Certificate Signing 
Request" but rather:

"Server failed request, will retry: 4301 (RPC failed at server.  Certificate 
operation cannot be completed: EXCEPTION (You did not provide a valid certificate for 
this operation))."

for two certs, and:

"Server failed request, will retry: 907 (RPC failed at server.  cannot connect to 
'https://HOSTNAME.DOMAIN:443/ca/agent/ca/displayBySerial': [Errno -8053] (SEC_ERROR_BUSY) 
NSS could not shutdown. Objects are still in use.)."

for a third.

After some time, I resubmit and the error returned changes to "Failure 
decoding..." for all three (expired) certs.

In the httpd error_log during that time, I see the following errors and 
traceback:

[Sun Oct 06 21:13:14 2013] [error] /usr/lib64/python2.6/getpass.py:83: 
GetPassWarning: Can not control echo on the terminal.
[Sun Oct 06 21:13:14 2013] [error]   passwd = fallback_getpass(prompt, stream)
[Sun Oct 06 21:13:14 2013] [error] Warning: Password input may be echoed.
[Sun Oct 06 21:13:14 2013] [error] Enter password for internal:
[Sun Oct 06 21:13:14 2013] [error] exception in PK11 password callback
[Sun Oct 06 21:13:14 2013] [error] Traceback (most recent call last):
[Sun Oct 06 21:13:14 2013] [error]   File 
"/usr/lib/python2.6/site-packages/ipapython/nsslib.py", line 229, in 
password_callback
[Sun Oct 06 21:13:14 2013] [error] return getpass.getpass("Enter password for 
%s: " % slot.token_name);
[Sun Oct 06 21:13:14 2013] [error]   File "/usr/lib64/python2.6/getpass.py", 
line 83, in unix_getpass
[Sun Oct 06 21:13:14 2013] [error] passwd = fallback_getpass(prompt, stream)
[Sun Oct 06 21:13:14 2013] [error]   File "/usr/lib64/python2.6/getpass.py", 
line 118, in fallback_getpass
[Sun Oct 06 21:13:14 2013] [error] return _raw_input(prompt, stream)
[Sun Oct 06 21:13:14 2013] [error]   File "/usr/lib64/python2.6/getpass.py", 
line 135, in _raw_input
[Sun Oct 06 21:13:14 2013] [error] raise EOFError
[Sun Oct 06 21:13:14 2013] [error] EOFError

It looks like perhaps there is a

Re: [Freeipa-users] Failure decoding Certificate Signing Request

2013-10-23 Thread Thomson, Ryan
> -Original Message-
> From: Rob Crittenden [mailto:rcrit...@redhat.com]
> Sent: Tuesday, October 22, 2013 7:13 PM
> To: Thomson, Ryan; freeipa-users@redhat.com
> Subject: Re: [Freeipa-users] Failure decoding Certificate Signing Request
> 
> Thomson, Ryan wrote:
> >> -Original Message-
> >> From: Rob Crittenden [mailto:rcrit...@redhat.com]
> >> Sent: Tuesday, October 22, 2013 10:46 AM
> >> To: Thomson, Ryan; freeipa-users@redhat.com
> >> Subject: Re: [Freeipa-users] Failure decoding Certificate Signing
> >> Request
> >>
> >> Thomson, Ryan wrote:
> >>> Hi Rob,
> >>>
> >>>> There is some duplication in the error strings (ticket
> >>>> https://fedorahosted.org/freeipa/ticket/3988). Did you add a number
> >>>> prefix to yours, I see a 3 -in the error. If so, by my calculation,
> >>>> this works out to be an NSPRError. It would be helpful to know what
> >>>> exception is being raised, which we don't do.
> >>>
> >>> I did prefix numbers to the various error strings.
> >>>
> >>>> Either way, if you could enhance each occurrence of 'Failure
> >>>> decoding Certificate Signing Request' in /usr/lib/python*/site-
> >>>> packages/ipalib/plugins/cert.py to something like:
> >>>>
> >>>> except NSPEError, nsprerr:
> >>>>raise  errors.CertificateOperationError(error=_('Failure
> >>>> decoding Certificate Signing Request" %s') % nsprerr)
> >>>>
> >>>> You'll need to restart the httpd process afterwards. This should
> >>>> give us the real reason for the failure.
> >>>
> >>> Done. The error I get now is:
> >>>
> >>> Server failed request, will retry: 4301 (RPC failed at server.
> >>> Certificate
> >> operation cannot be completed: Failure decoding Certificate Signing
> >> Request" [Errno -8018] error (-8018) unknown).
> >>
> >> Hmm, very strange indeed.
> >>
> >> It should be using the NSS database initialized in mod_nss for
> >> Apache, which should remain open and available for wsgi. It almost
> >> seems like the database has been shut down.
> >>
> >> Can you add a logging event to log the value of nss.nss_is_initialized()?
> >>
> >> Have you done any configuration customization in Apache or mod_nss?
> >>
> >> thanks
> >>
> >> rob
> >
> > The return value of nss.nss_is_initialized() is False when I resubmit the
> (expired) certs through certmonger.
> 
> Ok, that is the core of the issue then. pkcs10.load_certificate() will 
> initialize
> NSS If it isn't already and I'm guessing that is failing and is the source of 
> this
> exception.
> 
> > I did have a custom config for apache that configured a virtual host with
> SSL. I have disabled that config and restarted httpd, resubmitted the certs to
> certmonger but I still receive the same error. I will continue poking through
> my apache / mod_nss config to see if anything stands out.
> 
> You're still using mod_nss though, right?
> 
> rob

I'm still using mod_nss.

I have discovered that I might be focusing on a symptom here rather than the 
core problem. If I restart httpd and then certmonger, the first error returned 
when certmonger tries to renew the certificates is not "Failure decoding 
Certificate Signing Request" but rather:

"Server failed request, will retry: 4301 (RPC failed at server.  Certificate 
operation cannot be completed: EXCEPTION (You did not provide a valid 
certificate for this operation))."

for two certs, and:

"Server failed request, will retry: 907 (RPC failed at server.  cannot connect 
to 'https://HOSTNAME.DOMAIN:443/ca/agent/ca/displayBySerial': [Errno -8053] 
(SEC_ERROR_BUSY) NSS could not shutdown. Objects are still in use.)."

for a third.

After some time, I resubmit and the error returned changes to "Failure 
decoding..." for all three (expired) certs.

In the httpd error_log during that time, I see the following errors and 
traceback:

[Sun Oct 06 21:13:14 2013] [error] /usr/lib64/python2.6/getpass.py:83: 
GetPassWarning: Can not control echo on the terminal.
[Sun Oct 06 21:13:14 2013] [error]   passwd = fallback_getpass(prompt, stream)
[Sun Oct 06 21:13:14 2013] [error] Warning: Password input may be echoed.
[Sun Oct 06 21:13:14 2013] [error] Enter password for internal: 
[Sun Oct 06 21:13:14 2013] [error] exception in PK11 password callback
[Sun Oct 06 21:13:14 2013] [er

Re: [Freeipa-users] Failure decoding Certificate Signing Request

2013-10-22 Thread Rob Crittenden

Thomson, Ryan wrote:

-Original Message-
From: Rob Crittenden [mailto:rcrit...@redhat.com]
Sent: Tuesday, October 22, 2013 10:46 AM
To: Thomson, Ryan; freeipa-users@redhat.com
Subject: Re: [Freeipa-users] Failure decoding Certificate Signing Request

Thomson, Ryan wrote:

Hi Rob,


There is some duplication in the error strings (ticket
https://fedorahosted.org/freeipa/ticket/3988). Did you add a number
prefix to yours, I see a 3 -in the error. If so, by my calculation,
this works out to be an NSPRError. It would be helpful to know what
exception is being raised, which we don't do.


I did prefix numbers to the various error strings.


Either way, if you could enhance each occurrence of 'Failure decoding
Certificate Signing Request' in /usr/lib/python*/site-
packages/ipalib/plugins/cert.py to something like:

except NSPEError, nsprerr:
   raise  errors.CertificateOperationError(error=_('Failure
decoding Certificate Signing Request" %s') % nsprerr)

You'll need to restart the httpd process afterwards. This should give
us the real reason for the failure.


Done. The error I get now is:

Server failed request, will retry: 4301 (RPC failed at server.  Certificate

operation cannot be completed: Failure decoding Certificate Signing
Request" [Errno -8018] error (-8018) unknown).

Hmm, very strange indeed.

It should be using the NSS database initialized in mod_nss for Apache, which
should remain open and available for wsgi. It almost seems like the database
has been shut down.

Can you add a logging event to log the value of nss.nss_is_initialized()?

Have you done any configuration customization in Apache or mod_nss?

thanks

rob


The return value of nss.nss_is_initialized() is False when I resubmit the 
(expired) certs through certmonger.


Ok, that is the core of the issue then. pkcs10.load_certificate() will 
initialize NSS If it isn't already and I'm guessing that is failing and 
is the source of this exception.



I did have a custom config for apache that configured a virtual host with SSL. 
I have disabled that config and restarted httpd, resubmitted the certs to 
certmonger but I still receive the same error. I will continue poking through 
my apache / mod_nss config to see if anything stands out.


You're still using mod_nss though, right?

rob

___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users


Re: [Freeipa-users] Failure decoding Certificate Signing Request

2013-10-22 Thread Thomson, Ryan
> -Original Message-
> From: Rob Crittenden [mailto:rcrit...@redhat.com]
> Sent: Tuesday, October 22, 2013 10:46 AM
> To: Thomson, Ryan; freeipa-users@redhat.com
> Subject: Re: [Freeipa-users] Failure decoding Certificate Signing Request
> 
> Thomson, Ryan wrote:
> > Hi Rob,
> >
> >> There is some duplication in the error strings (ticket
> >> https://fedorahosted.org/freeipa/ticket/3988). Did you add a number
> >> prefix to yours, I see a 3 -in the error. If so, by my calculation,
> >> this works out to be an NSPRError. It would be helpful to know what
> >> exception is being raised, which we don't do.
> >
> > I did prefix numbers to the various error strings.
> >
> >> Either way, if you could enhance each occurrence of 'Failure decoding
> >> Certificate Signing Request' in /usr/lib/python*/site-
> >> packages/ipalib/plugins/cert.py to something like:
> >>
> >> except NSPEError, nsprerr:
> >>   raise  errors.CertificateOperationError(error=_('Failure
> >> decoding Certificate Signing Request" %s') % nsprerr)
> >>
> >> You'll need to restart the httpd process afterwards. This should give
> >> us the real reason for the failure.
> >
> > Done. The error I get now is:
> >
> > Server failed request, will retry: 4301 (RPC failed at server.  Certificate
> operation cannot be completed: Failure decoding Certificate Signing
> Request" [Errno -8018] error (-8018) unknown).
> 
> Hmm, very strange indeed.
> 
> It should be using the NSS database initialized in mod_nss for Apache, which
> should remain open and available for wsgi. It almost seems like the database
> has been shut down.
> 
> Can you add a logging event to log the value of nss.nss_is_initialized()?
> 
> Have you done any configuration customization in Apache or mod_nss?
> 
> thanks
> 
> rob

The return value of nss.nss_is_initialized() is False when I resubmit the 
(expired) certs through certmonger.

I did have a custom config for apache that configured a virtual host with SSL. 
I have disabled that config and restarted httpd, resubmitted the certs to 
certmonger but I still receive the same error. I will continue poking through 
my apache / mod_nss config to see if anything stands out.

Cheers,

--Ryan

___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users


Re: [Freeipa-users] Failure decoding Certificate Signing Request

2013-10-22 Thread Rob Crittenden

Thomson, Ryan wrote:

Hi Rob,


There is some duplication in the error strings (ticket
https://fedorahosted.org/freeipa/ticket/3988). Did you add a number prefix
to yours, I see a 3 -in the error. If so, by my calculation, this works out to 
be
an NSPRError. It would be helpful to know what exception is being raised,
which we don't do.


I did prefix numbers to the various error strings.


Either way, if you could enhance each occurrence of 'Failure decoding
Certificate Signing Request' in /usr/lib/python*/site-
packages/ipalib/plugins/cert.py to something like:

except NSPEError, nsprerr:
  raise  errors.CertificateOperationError(error=_('Failure decoding
Certificate Signing Request" %s') % nsprerr)

You'll need to restart the httpd process afterwards. This should give us the
real reason for the failure.


Done. The error I get now is:

Server failed request, will retry: 4301 (RPC failed at server.  Certificate 
operation cannot be completed: Failure decoding Certificate Signing Request" 
[Errno -8018] error (-8018) unknown).


Hmm, very strange indeed.

It should be using the NSS database initialized in mod_nss for Apache, 
which should remain open and available for wsgi. It almost seems like 
the database has been shut down.


Can you add a logging event to log the value of nss.nss_is_initialized()?

Have you done any configuration customization in Apache or mod_nss?

thanks

rob

___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users


Re: [Freeipa-users] Failure decoding Certificate Signing Request

2013-10-22 Thread Thomson, Ryan
Hi Rob,

> There is some duplication in the error strings (ticket
> https://fedorahosted.org/freeipa/ticket/3988). Did you add a number prefix
> to yours, I see a 3 -in the error. If so, by my calculation, this works out 
> to be
> an NSPRError. It would be helpful to know what exception is being raised,
> which we don't do.

I did prefix numbers to the various error strings. 

> Either way, if you could enhance each occurrence of 'Failure decoding
> Certificate Signing Request' in /usr/lib/python*/site-
> packages/ipalib/plugins/cert.py to something like:
> 
> except NSPEError, nsprerr:
>  raise  errors.CertificateOperationError(error=_('Failure decoding
> Certificate Signing Request" %s') % nsprerr)
> 
> You'll need to restart the httpd process afterwards. This should give us the
> real reason for the failure.

Done. The error I get now is:

Server failed request, will retry: 4301 (RPC failed at server.  Certificate 
operation cannot be completed: Failure decoding Certificate Signing Request" 
[Errno -8018] error (-8018) unknown).

and in /var/log/httpd/error_log:

[Sat Oct 05 17:51:41 2013] [error] ipa: DEBUG: WSGI wsgi_dispatch.__call__:
[Sat Oct 05 17:51:41 2013] [error] ipa: DEBUG: WSGI xmlserver.__call__:
[Sat Oct 05 17:51:41 2013] [error] ipa: DEBUG: Created connection context.ldap2
[Sat Oct 05 17:51:41 2013] [error] ipa: DEBUG: WSGI WSGIExecutioner.__call__:
[Sat Oct 05 17:51:41 2013] [error] ipa: DEBUG: raw: 
cert_request(u'MIIDcDCCAlgCAQAwNjEUMBIGA1UEChMLRk1SSS5VQkMuQ0ExHjAcBgNVBAMTFXNoYW1yb2NrLmJyYWluLnViYy5jYTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKCzypT3oNmPx90Tevn/vv8FUouT8UL2d8qmhxK0AUVPxJwoZPtkbQBWzUNxkTBkhWV/5s0hN19VBb5ruHTbeSv7KBX8P+CwopQbbjpaqpwvi3dso1NSnT3kU+cNYY1j4tvyKkwPVS7FrP4oELX+aEEOuGF8eyOPK78UlZtDrY0Npje5l8MsUrRMKqQAjhIFc4EzTb2tqcR8Ch+OzBHugcFXcmXGmFnHkK29z2f7Aq1ynk0SqWC0r7nZXw/17jI1OEeD9pagGH1OLEzMrJUQTrvQGH/W+XPt2+ZvJ3UtF4ltw2ViStiG958b32OQvGnbQVJjaIgjpOSiorfnhM0wCPcCAwEAAaCB9DAaBgkqhkiG9w0BCRQxDRMLU2VydmVyLUNlcnQwgdUGCSqGSIb3DQEJDjGBxzCBxDAOBgNVHQ8BAQAEBAMCBPAwgZkGA1UdEQEBAASBjjCBi6A8BgorBgEEAYI3FAIDoC4MLGRvZ3RhZ2xkYXAvc2hhbXJvY2suYnJhaW4udWJjLmNhQEZNUkkuVUJDLkNBoEsGBisGAQUCAqBBMD+gDRsLRk1SSS5VQkMuQ0GhLjAsoAMCAQGhJTAjGwpkb2d0YWdsZGFwGxVzaGFtcm9jay5icmFpbi51YmMuY2EwFgYDVR0lAQEABAwwCgYIKwYBBQUHAwEwDQYJKoZIhvcNAQELBQADggEBAIP+0+O/COqFwbqUCJ+LJiE8aAmP01SRvfJx/RSE9huquDd2XdHVIQ6lQj6qnQYTtCw2NKRq77R3VmUAiCMpQw!
 
I9/x/QaaI4MBvV9iYA8b1H/weyvZAMw1mGkdgY55KWPhBtCqLuxHcGRblrtsy2PGp9wm/834s5YamQky+InQFlDy4o5ox+5U5iZS+pvKm52d0TQTozvZ/gSTAqEc+gpwlGAU4U0VaC+7PYnwkYJ98jLDaALm2OCWnSvw/02NLlc+h02mTjumAQ/YnWYNXiAtUbiA8BAkjT0UGV79Vi/aUKxpBTZQXbldrHN/cAmUtSMxebNNQjyUdzAHEV+TUUP2o=',
 principal=u'dogtagldap/HOSTNAME.DOMAIN@FULLY.QUALIFIED.DOMAIN', add=True)
[Sat Oct 05 17:51:41 2013] [error] ipa: DEBUG: 
cert_request(u'MIIDcDCCAlgCAQAwNjEUMBIGA1UEChMLRk1SSS5VQkMuQ0ExHjAcBgNVBAMTFXNoYW1yb2NrLmJyYWluLnViYy5jYTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKCzypT3oNmPx90Tevn/vv8FUouT8UL2d8qmhxK0AUVPxJwoZPtkbQBWzUNxkTBkhWV/5s0hN19VBb5ruHTbeSv7KBX8P+CwopQbbjpaqpwvi3dso1NSnT3kU+cNYY1j4tvyKkwPVS7FrP4oELX+aEEOuGF8eyOPK78UlZtDrY0Npje5l8MsUrRMKqQAjhIFc4EzTb2tqcR8Ch+OzBHugcFXcmXGmFnHkK29z2f7Aq1ynk0SqWC0r7nZXw/17jI1OEeD9pagGH1OLEzMrJUQTrvQGH/W+XPt2+ZvJ3UtF4ltw2ViStiG958b32OQvGnbQVJjaIgjpOSiorfnhM0wCPcCAwEAAaCB9DAaBgkqhkiG9w0BCRQxDRMLU2VydmVyLUNlcnQwgdUGCSqGSIb3DQEJDjGBxzCBxDAOBgNVHQ8BAQAEBAMCBPAwgZkGA1UdEQEBAASBjjCBi6A8BgorBgEEAYI3FAIDoC4MLGRvZ3RhZ2xkYXAvc2hhbXJvY2suYnJhaW4udWJjLmNhQEZNUkkuVUJDLkNBoEsGBisGAQUCAqBBMD+gDRsLRk1SSS5VQkMuQ0GhLjAsoAMCAQGhJTAjGwpkb2d0YWdsZGFwGxVzaGFtcm9jay5icmFpbi51YmMuY2EwFgYDVR0lAQEABAwwCgYIKwYBBQUHAwEwDQYJKoZIhvcNAQELBQADggEBAIP+0+O/COqFwbqUCJ+LJiE8aAmP01SRvfJx/RSE9huquDd2XdHVIQ6lQj6qnQYTtCw2NKRq77R3VmUAiCMpQwI9/x/!
 
QaaI4MBvV9iYA8b1H/weyvZAMw1mGkdgY55KWPhBtCqLuxHcGRblrtsy2PGp9wm/834s5YamQky+InQFlDy4o5ox+5U5iZS+pvKm52d0TQTozvZ/gSTAqEc+gpwlGAU4U0VaC+7PYnwkYJ98jLDaALm2OCWnSvw/02NLlc+h02mTjumAQ/YnWYNXiAtUbiA8BAkjT0UGV79Vi/aUKxpBTZQXbldrHN/cAmUtSMxebNNQjyUdzAHEV+TUUP2o=',
 principal=u'dogtagldap/HOSTNAME.DOMAIN@FULLY.QUALIFIED.DOMAIN', 
request_type=u'pkcs10', add=True)
[Sat Oct 05 17:51:41 2013] [error] ipa: INFO: 
host/HOSTNAME.DOMAIN@FULLY.QUALIFIED.DOMAIN: 
cert_request(u'MIIDcDCCAlgCAQAwNjEUMBIGA1UEChMLRk1SSS5VQkMuQ0ExHjAcBgNVBAMTFXNoYW1yb2NrLmJyYWluLnViYy5jYTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKCzypT3oNmPx90Tevn/vv8FUouT8UL2d8qmhxK0AUVPxJwoZPtkbQBWzUNxkTBkhWV/5s0hN19VBb5ruHTbeSv7KBX8P+CwopQbbjpaqpwvi3dso1NSnT3kU+cNYY1j4tvyKkwPVS7FrP4oELX+aEEOuGF8eyOPK78UlZtDrY0Npje5l8MsUrRMKqQAjhIFc4EzTb2tqcR8Ch+OzBHugcFXcmXGmFnHkK29z2f7Aq1ynk0SqWC0r7nZXw/17jI1OEeD9pagGH1OLEzMrJUQTrvQGH/W+XPt2+ZvJ3UtF4ltw2ViStiG958b32OQvGnbQVJjaIgjpOSiorfnhM0wCPcCAwEAAaCB9DAaBgkqhkiG9w0BCRQxDRMLU2VydmVyLUNlcnQwgdUGCSqGSIb3DQEJDjGBxzCBxDAOBgNVHQ8BAQAEBAMCBPAwgZkGA1UdEQEBAASBjjCBi6A8BgorBgEEAYI3FAIDoC4MLGRvZ3RhZ2xkYXAvc2hhbXJvY2suYnJhaW4udWJjLmNhQEZNUkkuVUJDLkNBoEsGBisGAQUCAqBBMD+gDRsLRk1SSS5VQkMuQ0GhLjAsoAMCAQGhJTAjGwpkb2d0YWdsZGFwGxVzaGFtcm9jay5icmFpbi51YmMuY2EwFgYDVR0lAQEABAwwCgYIKwYBBQUHAwEwDQYJK

Re: [Freeipa-users] Failure decoding Certificate Signing Request

2013-10-22 Thread Rob Crittenden

Thomson, Ryan wrote:

Hello FreeIPA users and developers,

I'm facing a problem with expired certificates in FreeIPA.

I have searched through the list archives and found advice to stop the IPA service, roll back the system 
clock to a time when the certs were valid but nearly expired, start IPA and then use certmonger to renew 
the certs as it normally should (getcert resubmit -i ). This appears to have worked as 
expected for the dogtag/CA certificates signed with the "dogtag-ipa-renew-agent" CA but is 
failing for the HTTP and dirsrv certificates signed with the "IPA" CA.

Certmonger reports the following error (ipa-getcert list):

Server failed request, will retry: 4301 (RPC failed at server.  Certificate 
operation cannot be completed: Failure decoding Certificate Signing Request).

I turned on IPA debugging to acquire the following error in 
/var/log/httpd/error_log when resubmitting to certmonger:

[Mon Oct 07 00:03:22 2013] [error] ipa: DEBUG: WSGI wsgi_dispatch.__call__:
[Mon Oct 07 00:03:22 2013] [error] ipa: DEBUG: WSGI xmlserver.__call__:
[Mon Oct 07 00:03:22 2013] [error] ipa: DEBUG: Created connection context.ldap2
[Mon Oct 07 00:03:22 2013] [error] ipa: DEBUG: WSGI WSGIExecutioner.__call__:
[Mon Oct 07 00:03:22 2013] [error] ipa: DEBUG: raw: 
cert_request(u'MIIDcDCCAlgCAQAwNjEUMBIGA1UEChMLRk1SSS5VQkMuQ0ExHjAcBgNVBAMTFXNoYW1yb2NrLmJyYWluLnViYy5jYTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKCzypT3oNmPx90Tevn/vv8FUouT8UL2d8qmhxK0AUVPxJwoZPtkbQBWzUNxkTBkhWV/5s0hN19VBb5ruHTbeSv7KBX8P+CwopQbbjpaqpwvi3dso1NSnT3kU+cNYY1j4tvyKkwPVS7FrP4oELX+aEEOuGF8eyOPK78UlZtDrY0Npje5l8MsUrRMKqQAjhIFc4EzTb2tqcR8Ch+OzBHugcFXcmXGmFnHkK29z2f7Aq1ynk0SqWC0r7nZXw/17jI1OEeD9pagGH1OLEzMrJUQTrvQGH/W+XPt2+ZvJ3UtF4ltw2ViStiG958b32OQvGnbQVJjaIgjpOSiorfnhM0wCPcCAwEAAaCB9DAaBgkqhkiG9w0BCRQxDRMLU2VydmVyLUNlcnQwgdUGCSqGSIb3DQEJDjGBxzCBxDAOBgNVHQ8BAQAEBAMCBPAwgZkGA1UdEQEBAASBjjCBi6A8BgorBgEEAYI3FAIDoC4MLGRvZ3RhZ2xkYXAvc2hhbXJvY2suYnJhaW4udWJjLmNhQEZNUkkuVUJDLkNBoEsGBisGAQUCAqBBMD+gDRsLRk1SSS5VQkMuQ0GhLjAsoAMCAQGhJTAjGwpkb2d0YWdsZGFwGxVzaGFtcm9jay5icmFpbi51YmMuY2EwFgYDVR0lAQEABAwwCgYIKwYBBQUHAwEwDQYJKoZIhvcNAQELBQADggEBAIP+0+O/COqFwbqUCJ+LJiE8aAmP01SRvfJx/RSE9huquDd2XdHVIQ6lQj6qnQYTtCw2NKRq77R3VmUAiCMp!

Qw!

  
I9/x/QaaI4MBvV9iYA8b1H/weyvZAMw1mGkdgY55KWPhBtCqLuxHcGRblrtsy2PGp9wm/834s5YamQky+InQFlDy4o5ox+5U5iZS+pvKm52d0TQTozvZ/gSTAqEc+gpwlGAU4U0VaC+7PYnwkYJ98jLDaALm2OCWnSvw/02NLlc+h02mTjumAQ/YnWYNXiAtUbiA8BAkjT0UGV79Vi/aUKxpBTZQXbldrHN/cAmUtSMxebNNQjyUdzAHEV+TUUP2o=',
 principal=u'dogtagldap/HOSTNAME.DOMAIN@FULLY.QUALIFIED.DOMAIN', add=True)
[Mon Oct 07 00:03:22 2013] [error] ipa: DEBUG: 
cert_request(u'MIIDcDCCAlgCAQAwNjEUMBIGA1UEChMLRk1SSS5VQkMuQ0ExHjAcBgNVBAMTFXNoYW1yb2NrLmJyYWluLnViYy5jYTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKCzypT3oNmPx90Tevn/vv8FUouT8UL2d8qmhxK0AUVPxJwoZPtkbQBWzUNxkTBkhWV/5s0hN19VBb5ruHTbeSv7KBX8P+CwopQbbjpaqpwvi3dso1NSnT3kU+cNYY1j4tvyKkwPVS7FrP4oELX+aEEOuGF8eyOPK78UlZtDrY0Npje5l8MsUrRMKqQAjhIFc4EzTb2tqcR8Ch+OzBHugcFXcmXGmFnHkK29z2f7Aq1ynk0SqWC0r7nZXw/17jI1OEeD9pagGH1OLEzMrJUQTrvQGH/W+XPt2+ZvJ3UtF4ltw2ViStiG958b32OQvGnbQVJjaIgjpOSiorfnhM0wCPcCAwEAAaCB9DAaBgkqhkiG9w0BCRQxDRMLU2VydmVyLUNlcnQwgdUGCSqGSIb3DQEJDjGBxzCBxDAOBgNVHQ8BAQAEBAMCBPAwgZkGA1UdEQEBAASBjjCBi6A8BgorBgEEAYI3FAIDoC4MLGRvZ3RhZ2xkYXAvc2hhbXJvY2suYnJhaW4udWJjLmNhQEZNUkkuVUJDLkNBoEsGBisGAQUCAqBBMD+gDRsLRk1SSS5VQkMuQ0GhLjAsoAMCAQGhJTAjGwpkb2d0YWdsZGFwGxVzaGFtcm9jay5icmFpbi51YmMuY2EwFgYDVR0lAQEABAwwCgYIKwYBBQUHAwEwDQYJKoZIhvcNAQELBQADggEBAIP+0+O/COqFwbqUCJ+LJiE8aAmP01SRvfJx/RSE9huquDd2XdHVIQ6lQj6qnQYTtCw2NKRq77R3VmUAiCMpQwI9/!

x/!

  
QaaI4MBvV9iYA8b1H/weyvZAMw1mGkdgY55KWPhBtCqLuxHcGRblrtsy2PGp9wm/834s5YamQky+InQFlDy4o5ox+5U5iZS+pvKm52d0TQTozvZ/gSTAqEc+gpwlGAU4U0VaC+7PYnwkYJ98jLDaALm2OCWnSvw/02NLlc+h02mTjumAQ/YnWYNXiAtUbiA8BAkjT0UGV79Vi/aUKxpBTZQXbldrHN/cAmUtSMxebNNQjyUdzAHEV+TUUP2o=',
 principal=u'dogtagldap/HOSTNAME.DOMAIN@FULLY.QUALIFIED.DOMAIN', 
request_type=u'pkcs10', add=True)
[Mon Oct 07 00:03:22 2013] [error] ipa: INFO: 
host/HOSTNAME.DOMAIN@FULLY.QUALIFIED.DOMAIN: 
cert_request(u'MIIDcDCCAlgCAQAwNjEUMBIGA1UEChMLRk1SSS5VQkMuQ0ExHjAcBgNVBAMTFXNoYW1yb2NrLmJyYWluLnViYy5jYTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKCzypT3oNmPx90Tevn/vv8FUouT8UL2d8qmhxK0AUVPxJwoZPtkbQBWzUNxkTBkhWV/5s0hN19VBb5ruHTbeSv7KBX8P+CwopQbbjpaqpwvi3dso1NSnT3kU+cNYY1j4tvyKkwPVS7FrP4oELX+aEEOuGF8eyOPK78UlZtDrY0Npje5l8MsUrRMKqQAjhIFc4EzTb2tqcR8Ch+OzBHugcFXcmXGmFnHkK29z2f7Aq1ynk0SqWC0r7nZXw/17jI1OEeD9pagGH1OLEzMrJUQTrvQGH/W+XPt2+ZvJ3UtF4ltw2ViStiG958b32OQvGnbQVJjaIgjpOSiorfnhM0wCPcCAwEAAaCB9DAaBgkqhkiG9w0BCRQxDRMLU2VydmVyLUNlcnQwgdUGCSqGSIb3DQEJDjGBxzCBxDAOBgNVHQ8BAQAEBAMCBPAwgZkGA1UdEQEBAASBjjCBi6A8BgorBgEEAYI3FAIDoC4MLGRvZ3RhZ2xkYXAvc2hhbXJvY2suYnJhaW4udWJjLmNhQEZNUkkuVUJDLkNBoEsGBisGAQUCAqBBMD+gDRsLRk1SSS5VQkMuQ0GhLjAsoAMCAQGhJTAjGwpkb2d0YWdsZGFwGxVzaGFtcm9jay5icmFpbi51YmMuY2EwFgYDVR0lAQEABAwwCgYIKwYBBQUHAwEwDQYJKoZIhvcNAQELBQADggEBAIP+0+O/COqFwbqUCJ+LJiE8aAmP01SRvfJx/RSE9huqu!

Dd!

  
2XdHVIQ6lQj6qnQYTtCw2NKRq77R3VmUAiCMpQwI9/x/QaaI4MBvV9iYA8b1H/weyvZAMw1mGkdgY55KWPh